Your browser doesn't support JavaScript. Please upgrade to a modern browser or enable JavaScript in your existing browser.
Skip Navigation U.S. Department of Health and Human Services www.hhs.gov
Agency for Healthcare Research Quality www.ahrq.gov
Archive print banner

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: https://info.ahrq.gov. Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to www.ahrq.gov for current information.

Developing the instrument and process for abstracting data

To systematically review Cost-effectiveness analyses (CEAs), we first developed a tool for abstracting relevant information from individual studies in a standardized way. Through a literature search and consultation with experts, we located several existing abstraction tools (28,39) and quality-rating criteria (4,40,41). We drew on the strengths of each of these instruments to develop a novel abstraction tool that met the needs and objectives of the USPSTF. The tool is largely adapted from the CEA abstraction instruments developed by the Task Force on Community Preventive Services (39) and the Harvard Center for Risk Analysis (29).

The USPSTF CEA abstraction instrument, which we are currently piloting, was designed to (1) ensure that the CEAs being reviewed are applicable to the question posed, (2) assess the studies' methodological rigor, (3) ascertain that their models contain the appropriate components of effectiveness, (4) examine the degree to which they use the best available evidence of effectiveness, (5) evaluate the validity and impact of their assumptions, and (6) assess the type and quality of cost information used. Our criteria for rating methodological quality are based on recommendations of the Panel on Cost-Effectiveness in Health and Medicine (4). Sample quality rating items are listed in Table 2. Because simple counts of criteria fulfilled do not differentiate studies of varying quality with great precision, these criteria will not be used to generate quality scores but rather as a guide in categorizing study quality as high, fair, or poor.

Table 2. Sample quality rating items for cost-effectiveness analyses

Framing

  • Are the interventions and populations compared appropriate?
  • Is the study conducted from the societal perspective?
  • Is the time horizon clinically appropriate and relevant to the study question?

Effects

  • Are all important drivers of effectiveness included?
  • Are key harms included?
  • Is the best available evidence used to estimate effectiveness?
  • Are long-term outcomes used?
  • Do effect measures capture preferences or utilities?

Costs

  • Are all appropriate downstream medical costs included?
  • Are charges converted to costs appropriately?
  • Are the best available data used to estimate costs?

Results

  • Are incremental cost-effectiveness ratios presented?
  • Are appropriate sensitivity analyses performed?

Our reviews will focus mainly on high-quality studies. When few or no high-quality studies exist for a given question, we will also consider studies rated fair. Poor-quality studies will not be considered. Our objective is to present the best evidence currently available while appropriately identifying study limitations, because policymakers are sometimes required to make decisions without having perfect information.

Return to Contents

Process for reviewing studies

The process for systematically reviewing CEAs is similar to that for reviewing studies of effectiveness (Table 3). As with any systematic review, before reviewing the evidence, one must define the question at hand. USPSTF "topic teams" (38) reviewing the evidence for the effectiveness of preventive services within specific clinical topics (e.g., screening for hypertension) are asked to identify relevant questions related to the cost-effectiveness of services within each topic (Table 1).

Table 3. U.S. Preventive Services Task Force process for conducting a systematic review of cost-effectiveness analyses

  1. Define the question to be addressed.
  2. Comprehensively search relevant literature databases.
  3. Screen abstracts for inclusion.
  4. Review reference lists and call experts to identify studies not captured by the literature search.
  5. Abstract relevant studies.
  6. Compare the impact of varying assumptions and resolve differential results across studies through consensus.
  7. Synthesize and present results in evidence tables.

Once the question is identified, a comprehensive search for appropriate CEAs is conducted. Searches may be limited by year (e.g., after 1990), based on when relevant technologies came into use. Abstracts identified by the search are screened for inclusion using three items: (1) Does the study address the identified question? (2) Is the study an original CEA? (3) Does the study report results using an appropriate outcome metric (e.g., LYS, QALY, or cases of illness averted)? If no CEAs exist for a specific question, we consider reviewing other types of economic analyses, such as CBAs.

Studies meeting inclusion criteria are abstracted by at least two reviewers. To determine whether the best available evidence was used in each study, whether included assumptions are reasonable, and whether each study appropriately addresses the question at hand, the CEA review team for each question includes a member of the topic team reviewing the effectiveness evidence for that topic (38). The data used in the CEA are compared with the evidence derived from the systematic review of effectiveness conducted by the topic team, which serves as the "gold standard" for whether the best available evidence was used.

After abstracting studies, reviewers discuss how studies differ in their assumptions, how varying assumptions affect study results, and how different studies may arrive at different conclusions. Finally, key information addressing the initial question and highlighting study quality and the effect of various assumptions are summarized in evidence tables.

Return to Contents

Conclusions

CEAs are valuable tools for incorporating cost considerations into evidence-based clinical decisions. This article has outlined the USPSTF's strategy to incorporate information from CEAs into its process for recommending clinical preventive services. Through these efforts, we hope not only to provide guidance about implementing preventive services, but to identify unmet needs in economic analyses of preventive health care, illuminate some of the trade-offs in alternative approaches to delivering preventive services, and provide substrata for policy discussions and public debate over the role of cost-effectiveness in allocating health care resources.

In the future, ranking of preventive services based on cost-effectiveness may provide busy clinicians and their patients with some scientific basis for deciding how to best spend their limited time providing or carrying out the services that are most likely to have the greatest impact on health. At present, CEAs should be considered an important aid to decision makers striving to achieve the best possible health for a population.

Return to Contents

Acknowledgements

Supported by contract #290-97-0018 (Task Order #2) to the Oregon Health Sciences University (SS, MH) and contract #290-97-0011 (Task Order #3) to the Research Triangle Institute/University of North Carolina (TJH, MP) from the Agency for Healthcare Research and Quality, and contract # DAMD17-94-J-4212 from the Department of the Army (JSM). The views expressed in this article are those of the authors and do not necessarily reflect those of the Agency for Healthcare Research and Quality or the Department of the Army.

We thank Vilma G. Carande-Kulis, M.S., Ph.D.; Joanna E. Siegel, Sc.D., R.N.; Dennis G. Fryback, Ph.D.; Milton J. Weinstein, Ph.D.; and the members of the third U.S. Preventive Services Task Force, for comments on earlier versions of this manuscript; the Partnership for Prevention Committee on Clinical Preventive Service Priorities and Peter J. Neumann, Sc.D., for sharing their work; and Kathryn Pyle Krages, AMLS, M.A., Susan Wingenfeld, and Gary Miranda, M.A., for administrative support and help with manuscript preparation and editing.

Return to Contents

References and Notes

1. Levit K, Cowan C, Lazenby H, et al. Health spending in 1998: signals of change. The Health Accounts Team. Health Affairs 2000, 19:124-132.

2. Weinstein MJ, Garber AM, Fryback DG. Disease Prevention Research at NIH: An Agenda for All. Workshop J: Prevention strategies, economic realities, and identification of prevention research needs. Prev Med 1994, 23:571-572.

3. Teutsch SM, Murray JF. Dissecting cost-effectiveness analysis for preventive interventions: a guide for decision makers. Am J Manag Care 1999, 5:301-305.

4. Gold M, Seigel J, Russell L, Weinstein MC. Cost-effectiveness in health and medicine. New York: Oxford University Press, 1996.

5. Gold MR, Patrick DL, Torrance GW, et al. Identifying and valuing outcomes. In: Gold M, Seigel J, Russell L, Weinstein MC, eds. Cost-effectiveness in health and medicine. New York: Oxford University Press, 1996:82-134.

6. Kattlove H, Liberati A, Keeler E, Brook RH. Benefits and costs of screening and treatment for early breast cancer: development of a basic benefit package. JAMA 1995, 273:142-148.

7. Prosser LA, Koplan JP, Neumann PJ, Weinstein MC. Barriers to using cost-effectiveness analysis in managed care decision making. Am J Manag Care 2000, 6:173-179.

8. Mandel JS, Bond JH, Church TR, et al. Reducing mortality from colorectal cancer by screening for fecal occult blood: Minnesota Colon Cancer Control Study. New Eng J Med 1993, 328:1365-1371. (published erratum appears in N Engl J Med 1993 Aug 26;329[9]:672)

9. Selby JV, Friedman GD, Quesenberry CP Jr, Weiss NS. A case-control study of screening sigmoidoscopy and mortality from colorectal cancer. New Eng J Med 1992, 326:653-657.

10. Winawer SJ, Fletcher RH, Miller L, et al. Colorectal cancer screening: clinical guidelines and rationale. Gastroenterology 1997, 112:594-642. (published errata appear in Gastroenterology 1997;112[3]:1060 and 1998;114[3]:625).

11. Eddy DM. Screening for colorectal cancer. Ann Intern Med 1990, 113:373-384.

12. Frazier AL, Colditz GA, Fuchs CS, Kuntz KM. Cost-effectiveness of screening for colorectal cancer in the general population. JAMA 2000, 284:1954-1961.

13. Khandker RK, Dulski JD, Kilpatrick JB, Ellis RP, Mitchell JB, Baine WB. A decision model and cost-effectiveness analysis of colorectal cancer screening and surveillance guidelines for average-risk adults. Int J Technol Assess Health Care 2000, 16:799-810.

14. Wagner JL, Tunis S, Brown M, Ching A, Almeida R. Cost-effectiveness of colorectal cancer screening in average-risk adults. In: Young GP, Rozen P, Levin B, eds. Prevention and early detection of colorectal cancer. Philadelphia: W.B. Saunders, 1996:321-56.

15. Eddy DM. Screening for cervical cancer. Ann Intern Med 1990, 113:214-226.

16. Eddy DM. Screening for cervical cancer: common screening tests. Philadelphia: American College of Physicians, 1991.

17. Kerlikowske K, Salzmann P, Phillips KA, Cauley JA, Cummings SR. Continuing screening mammography in women aged 70 to 79 years: impact on life expectancy and cost-effectiveness. JAMA 1999, 282:2156-2163.

18. Prosser LA, Stinnett AA, Goldman PA, et al. Cost-effectiveness of cholesterol-lowering therapies according to selected patient characteristics. Ann Intern Med 2000, 132:769-779.

19. Mandelblatt JS, Fahs MC. The cost-effectiveness of cervical cancer screening for low-income elderly women. JAMA 1988, 259:2409-2413.

20. Fahs MC, Mandelblatt J, Schechter C, Muller C. cost-effectiveness of cervical cancer screening for the elderly. Ann Intern Med 1992, 117:520-527.

21. Ford D, Easton DF, Stratton M, et al. Genetic heterogeneity and penetrance analysis of the BRCA1 and BRCA2 genes in breast cancer families: the Breast Cancer Linkage Consortium. Am J Hum Genet 1998, 62:676-689.

22. Burke W, Daly M, Garber J, et al. Recommendations for follow-up care of individuals with an inherited predisposition to cancer. II. BRCA1 and BRCA2. Cancer Genetics Studies Consortium. JAMA 1997, 277:997-1003.

23. Schrag D, Kuntz KM, Garber JE, Weeks JC. Benefit of prophylactic mastectomy for women with BRCA1 or BRCA2 mutations. JAMA 2000;283:3070-3072.

24. Martin DH, Mroczkowski TF, Dalu ZA, et al. A controlled trial of a single dose of azithromycin for the treatment of chlamydial urethritis and cervicitis. The Azithromycin for Chlamydial Infections Study Group. New Eng J Med 1992, 327:921-925.

25. Genc M, Mardh A. A cost-effectiveness analysis of screening and treatment for Chlamydia trachomatis infection in asymptomatic women. Ann Intern Med 1996, 124:1-7.

26. Russell LB, Siegel JE, Daniels N, Gold MR, Luce BR, Mandelblatt J. Cost-effectiveness analysis as a guide to resource allocation in health: roles and limitations. In: Gold M, Seigel J, Russell L, Weinstein MC, eds. Cost-effectiveness in health and medicine. New York: Oxford University Press, 1996:3-24.

27. Drummond M, Torrance G, Mason J. Cost-effectiveness league tables: more harm than good? Soc Sci Med 1993, 37:33-40.

28. Stone PW, Teutsch SM, Chapman RH, Bell C, Goldie SJ, Neumann PJ. Cost-utility analyses of clinical preventive services: published ratios, 1976-1997. Am J Prev Med 2000, 19:15-23.

29. Neumann PJ, Stone PW, Chapman RH, Sandberg EA, Bell CM. The quality of reporting in published cost-utility analyses, 1976-1997. Ann Intern Med 2000, 132:964-972.

30. Russell LB, Gold MR, Siegel JE, Daniels N, Weinstein MC. The role of cost-effectiveness analysis in health and medicine. Panel on Cost-Effectiveness in Health and Medicine. JAMA 1996, 276:1172-1177.

31. Brown ML, Fintor L. Cost-effectiveness of breast cancer screening: preliminary results of a systematic review of the literature. Breast Cancer Res Treat 1993, 25:113-118.

32. Ubel PA, Nord E, Gold M, Menzel P, Prades JL, Richardson J. Improving value measurement in cost-effectiveness analysis. Med Care 2000, 38:892-901.

33. Daniels N. Rationing fairly: programmatic considerations. Bioethics 1993, 7:223-233.

34. Klevit HD, Bates AC, Castanares T, Kirk EP, Sipes-Metzler PR, Wopat R. Prioritization of health care services: a progress report by the Oregon Health Services Commission. Arch Intern Med 1991, 151:912-916.

35. Hadorn DC. Setting health care priorities in Oregon: cost-effectiveness meets the rule of rescue. JAMA 1991, 265:2218-2225.

36. Brown LD. The national politics of Oregon's rationing plan. Health Affairs 1991, 10:28-51.

37. Coffield AB, Maciosek MV, McGinnis JM, et al. Priorities among recommended clinical preventive services. Am J Prev Med 2001. In press.

38. Harris RP, Helfand M, Woolf SH, et al. Current methods of the U.S. Preventive Services Task Force: a review of the process. Am J Prev Med 2001;20(suppl 3):21-35.

39. Carande-Kulis VG, Maciosek MV, Briss PA, et al. Methods for systematic reviews of economic evaluations for the Guide to Community Preventive Services. Task Force on Community Preventive Services. Am J Prev Med 2000, 18:75-91.

40. Drummond MF, Jefferson TO. Guidelines for authors and peer reviewers of economic submissions to the BMJ. Br Med J 1996, 313:275-283.

41. Udvarhelyi IS, Colditz GA, Rai A, Epstein AM. Cost-effectiveness and cost-benefit analyses in the medical literature. Are the methods being used correctly? Ann Intern Med 1992, 116:238-244.

Author Affiliations

[a] Saha, Helfand: Evidence-Based Practice Center, Oregon Health Sciences University, Portland, OR.
[b] Helfand: Division of Medical Informatics and Outcomes Research, Oregon Health Sciences University, Portland, OR.
[c] Saha, Helfand: Section of General Internal Medicine, Portland Veterans Affairs Medical Center, Portland, OR.
[d] Hoerger, Pignone: Evidence-based Practice Center, Research Triangle Institute and University of North Carolina, Chapel Hill, NC.
[e] Hoerger: Center for Economics Research, Research Triangle Institute, Research Triangle Park, Chapel Hill, NC.
[f] Pignone: Department of Internal Medicine, University of North Carolina at Chapel Hill, Chapel Hill, NC.
[g] Teutsch: Merck & Co, Inc, West Point, PA.
[h] Mandelblatt: Departments of Oncology and Medicine, Georgetown University Medical Center, Washington, DC.

*Other members of the Cost Work Group of the third U.S. Preventive Services Task Force include: David Atkins, M.D, M.P.H, Agency for Healthcare Research and Quality, Rockville, MD; Alfred O. Berg, M.D, M.P.H, University of Washington, Seattle, WA; Tracy A. Lieu, M.D, M.P.H, Harvard Community Health Plan, Boston, MA; Cynthia D. Mulrow, M.D, M.Sc, University of Texas Health Science Center, San Antonio, TX; Harold C. Sox, Jr, Dartmouth-Hitchcock Medical Center, Lebanon, NH; and Carolyn Westhoff, M.D, M.Sc, Columbia University, New York, NY.

Copyright and Source Information

This document is in the public domain within the United States as stated in AHRQ's license agreement with the American Journal of Preventive Medicine. For information on reprinting, contact Randie Siegel, Director, Division of Printing and Electronic Publishing, Agency for Healthcare Research and Quality, Suite 501, 2101 East Jefferson Street, Rockville, MD 20852. Requests for linking or to incorporate content in electronic resources should be sent to: https://info.ahrq.gov.

Source: Saha S, Hoerger TJ, Pignone MP, Teutsch SM, Helfand M, Mandelblatt JS. The Art and Science of Incorporating cost-effectiveness into Evidence-based Recommendations for Clinical Preventive Services. Am J Prev Med 2001;20(3S):36-43 (http://www.elsevier.com/locate/online).

Return to Contents

 

The information on this page is archived and provided for reference purposes only.

 

AHRQ Advancing Excellence in Health Care