The Outcome of Outcomes Research at AHCPR: References
Barry MJ, Cherkin DC, Chang Y, et al. A randomized trial of a multimedia shared decision-making program for men facing a treatment decision for benign prostatic hypertrophy. Dis Management Clin Outcomes 1997;1:5-14.
Barry MJ, Fowler FJ, Chang Y, Liss CL, Wilson H, Stek M Jr. The American Urological Association symptom index: does mode of administration affect its psychometric properties? J Urol 1995 Sep;154(3):1056-9.
Barry MJ, Fowler FJ Jr, O'Leary MP, Bruskewitz RC, Holtgrewe HL, Mebust WK, Cockett AT. The American Urological Association symptom index for benign prostatic hyperplasia. The Measurement Committee of the American Urological Association. J Urol 1992 Nov;148(5):1549-57.
Barry MJ, Williford WO, Chang Y, Machi M, Jones KM, Walker-Corkery E, Lepor H. Benign prostatic hyperplasia specific health status measures in clinical research: how much change in the American Urological Association symptom index and the benign prostatic hyperplasia impact index is perceptible to patients? J Urol 1995 Nov;154(5):1770-4.
Barry MJ, Fowler FJ Jr, O'Leary MP, Bruskewitz RC, Holtgrewe HL, Mebust WK. Correlation of the American Urological Association symptom index with self-administered versions of the Madsen-Iversen, Boyarsky and Maine Medical Assessment Program symptom indexes. Measurement Committee of the American Urological Association. J Urol 1992 Nov;148(5):1558-63.
Barry MJ, Walker-Corkery E, Chang Y, Tyll LT, Cherkin DC, Fowler FJ. Measurement of overall and disease-specific health status: does the order of questionnaires make a difference? J Health Serv Res Policy 1996 Jan;1(1):20-7.
Barry MJ, Fowler FJ, O'Leary MP, Bruskewitz RC, Holtgrewe HL, Mebust WK. Measuring disease-specific health status in men with benign prostatic hyperplasia. Med Care 1995;33(4):AS145-AS155.
Barry MJ, Girman CJ, O'Leary MP, Walker-Corkery ES, Binkowitz BS, Cockett AT, Guess HA. Using repeated measures of symptom score, uroflowmetry and prostate specific antigen in the clinical management of prostate disease. Benign Prostatic Hyperplasia Treatment Outcomes Study Group. J Urol 1995 Jan;153(1):99-103.
Barry MJ, Mulley AG Jr., Fowler FJ, et al. Watchful waiting vs. immediate transurethral resection for symptomatic prostatism. The importance of patients' preferences. JAMA 1988; 259(20):3010-17.
Bates DW, Cullen DJ, Laird N, Petersen LA, Small SD, Servi D, Laffel G, Sweitzer BJ, Shea BF, Hallisey R, et al. Incidence of adverse drug events and potential adverse drug events. Implications for prevention. ADE Prevention Study Group. JAMA 1995 Jul 5;274(1):29-34.
Bates DW, Spell N, Cullen DJ, Burdick E, Laird N, Petersen LA, Small SD, Sweitzer BJ, Leape LL. The costs of adverse drug events in hospitalized patients. Adverse Drug Events Prevention Study Group. JAMA 1997 Jan 22-29;277(4):307-11.
Berman S, Byrns PJ, Bondy J, Smith PJ, Lezotte D. Otitis media-related antibiotic prescribing patterns, outcomes, and expenditures in a pediatric Medicaid population. Pediatrics 1997;100(4):585-92.
Bombardier C, Melfi CA, Paul J, Green R, Hawker G, Wright J, Coyte P. Comparison of a generic and a disease-specific measure of pain and physical function after knee replacement surgery. Med Care 1995 Apr;33(4 Suppl):AS131-44.
Cassard SD, Patrick DL, Damiano AM, Legro MW, Tielsch JM, Diener-West M, Schein OD, Javitt JC, Bass EB, Steinberg EP. Reproducibility and responsiveness of the VF-14. An index of functional impairment in patients with cataracts. Arch Ophthalmol 1995 Dec;113(12):1508-13.
Chassin MR, Kosecoff J, Park RE, et al. Does inappropriate use explain geographic variations in the use of health care services? A study of three procedures. JAMA 1987;258:2533-7.
Clancy CM, Eisenberg JM. Outcomes research at the Agency for Health Care Policy and Research. Dis Man Clin Outcomes 1998; 3: 72-80.
Collins R, Julian D. British Heart Foundation Surveys of UK treatment policies for acute myocardial infarction (1987 and 1989). British Heart J 1991 Sep;66(3):250-5.
Davis DA, Thomson MA, Oxman AD, et al. Changing physician performance. A systematic review of the effect of continuing medical education strategies. JAMA 1995; 274(9):700-5.
Deyo RA, Patrick DL. The significance of treatment effects: the clinical perspective. Med Care 1995;33(Supplement):AS286-AS290.
Deyo RA. Promises and limitations of the Patient Outcome Research Teams: the low-back pain example. Proceedings of the Association of American Physicians 1995 Oct;107(3):324-8.
Eagle KA, Lee TH, Brennan TA, et al. 28th Bethesda Conference. Task Force 2: Guideline implementation. Journal of the American College of Cardiology1997 May;29(6):1141-8.
ECRI. 1999 Healthcare Standards Directory. ECRI, Plymouth Meeting, PA. 1998.
Eddy DM, Billings J. The quality of medical evidence: implications for quality of care. Health Aff 1998; 7(1):19-32.
Elixhauser A, Halpern M, Schmier J, et al. Health care CBA and CEA from 1991 to 1996: An updated bibliography. Med Care 1998 May;36(5 Suppl):MS1-9, MS18-147.
Fine MJ, Hanusa BH, Lave JR, Singer DE, Stone RA, Weissfeld LA, Coley CM, Marrie TJ, Kapoor WN. Comparison of a disease-specific and a generic severity of illness measure for patients with community-acquired pneumonia. J Gen Intern Med 1995 Jul;10(7):359-68.
Fine MJ, Singer DE, Marrie TJ, Lave JR, Coley CM, Schulz R, Rogers JC, Stone RA, Karpf M, Ricci EM, Obrosky DS, Hough LJ, Kapoor WN. Medical outcomes of ambulatory and hospitalized low risk patients with community-acquired pneumonia. J Gen Intern Med 1994;9(supplement 2):29A.
Fowler FJ Jr, Barry MJ. Quality of life assessment for evaluating benign prostatic hyperplasia treatments. An example of using a condition-specific index. Eur Urol 1993;24 Suppl 1:24-7.
Gill TM, Feinstein AR. A critical appraisal of the quality of quality-of-life measurements. JAMA 1994;272:619-26.
Goldenberg RL. Low Birthweight in Minority and High-Risk Women Patient Outcomes Research Team (PORT) Final Report. AHCPR Publication No. 98-N005. Rockville, MD. Agency for Health Care Policy and Research, 1998.
Gray BH. The legislative battle over health services research. Health Aff (Millwood) 1992 Winter;11(4):38-66.
Hannan EL, Racz M, Ryan TJ, McCallister BD, Johnson LW, Arani DT, Guerci AD, Sosa J, Topol EJ. Coronary angioplasty volume-outcome relationships for hospitals and cardiologists. JAMA 1997 Mar 19;277(11):892-8.
Hirshfeld JW Jr, Ellis SG, Faxon DP. Recommendations for the assessment and maintenance of proficiency in coronary interventional procedures: Statement of the American College of Cardiology. J Am Coll Cardiol 1998 Mar 1;31(3):722-43.
Hornberger J, Wrone E. When to base clinical policies on observational versus randomized trial data. Ann Intern Med 1997 Oct 15;127(8 Pt 2):697-703.
Hulley S, Grady D, Bush T, et al, for the HERS Research Group. Randomized trial of estrogen plus progestin for secondary prevention of coronary heart disease in postmenopausal women. JAMA 1998;280:605-13.
Javitt JC, Tielsch JM, Canner JK, et al. National outcomes of cataract extraction. Increased risk of retinal complications associated with Nd:YAG laser capsulotomy. The Cataract Patient Outcomes Research Team. Ophthalmology 1992;99(10):1487-98.
Jensen MP, Strom SE, Turner JA, Romano JM. Validity of the Sickness Impact Profile Roland scale as a measure of dysfunction in chronic pain patients. Pain 1992 Aug;50(2):157-62
Leape LL, Park RE, Solomon DH, et al. Does inappropriate use explain small-area variations in the use of health care services? JAMA 1990;263:669-72.
Lomas J. Retailing research: increasing the role of evidence in clinical services for childbirth. Milbank Q 1993;71(3):439-75.
McPherson K, Wennberg JE, Hovind OB, et al. Small-area variations in the use of common surgical procedures: an international comparison of New England, England, and Norway. N Engl J Med 1982;307:1310-4;
Mendelson DN, Goodman CS, Ahn R, et al. Outcomes and effectiveness research in the private sector. Health Aff 1998;17(5):75-90
O'Connor GT, Plume SK, Olmstead EM, Morton JR, Maloney CT, Nugent WC, Hernandez F Jr, Clough R, Leavitt BJ, Coffin LH, Marrin CA, Wennberg D, Birkmeyer JD, Charlesworth DC, Malenka DJ, Quinton HB, Kasper JF. A regional intervention to improve the hospital mortality associated with coronary artery bypass graft surgery. The Northern New England Cardiovascular Disease Study Group. JAMA 1996 Mar 20;275(11):841-6.
Office of Technology Assessment. Identifying health technologies that work: searching for evidence. U.S. Congress. 1994.
O'Leary MP, Fowler FJ, Lenderking WR, Barber B, Sagnier PP, Guess HA, Barry MJ. A brief male sexual function inventory for urology. Urology 1995 Nov;46(5):697-706.
Patrick DL, Deyo RA, Atlas SJ, Singer DE, Chapin A, Keller RB. Assessing health-related quality of life in patients with sciatica. Spine 1995 Sep 1;20(17):1899-908.
Petitti DB. Hormone replacement therapy and heart disease prevention: experimentation trumps observation. JAMA 1998;280:650-1.
Powe NR, Turner JA, Maklan CW, et al. Alternative Methods for formal literature review and meta-analysis in AHCPR patient outcomes research teams. Med Care 1994; 32(7):JS22-JS37.
Reinhardt U. The Social Perspective. In Heithoff K, Effectiveness and Outcomes in Health Care. National Academy Press. Ch. 6, 1990. Washington, DC.
Roper WL, Winkenwerder W, Hackbarth GM, et al. Effectiveness in health care. An initiative to evaluate and improve medical practice. N Engl J Med 1988;319(18):1197-202.
Shea JA, Healey MJ, Berlin JA, et al. Mortality and complications associated with laparoscopic cholecystectomy. A meta-analysis. Ann Surg 1996;224(5):609-20
Sheldon T. Please bypass the PORT. BMJ 1994 Jul 16;309(6948):142-43
Soumerai SB, McLaughlin TJ, Spiegelman D, et al. Adverse outcomes of underuse of beta-blockers in elderly survivors of acute myocardial infarction. JAMA 1997;277:115-21.
Steinberg EP, Tielsch JM, Schein OD, Javitt JC, Sharkey P, Cassard SD, Legro MW, Diener-West M, Bass EB, Damiano AM, et al. The VF-14. An index of functional impairment in patients with cataract. Arch Ophthalmol 1994;112(5)630-8.
Treasure T, MacRae KD. Minimisation: the platinum standard for trials? BMJ 1998;317:362-3.
Wennberg J. Dealing with medical practice variation: a proposal for action. Health Aff 1984 Summer;3:6-32.
Appendix 1: Survey of COER-Funded Principal Investigators: Summary
As one of the first activities in this OER critique, we felt it would be useful to have a systematic review of what had been produced so far with Agency funding. Therefore, a mailed survey was sent to all principal investigators (PIs) funded by COER between 1989 and 1997, asking them to describe their most important work. The survey was sent to all PIs who had participated in Patient Outcomes Research Teams, Minority Research Centers, Pharmaceutical Outcomes Therapy studies, and other recipients of COER funding. This letter asked PIs to "identify the most salient findings" from each of their outcomes/effectiveness research studies. They were asked to provide enough material for up to three slides, and to state findings in "succinct, bullet-point format." No specific criteria were provided to define the "salience" of findings, nor were PIs instructed about the number of findings to be submitted. The objective was to identify whether any common themes could be derived from a large set of research findings identified by PIs as being their most important results.
The survey letter was mailed to 95 PIs. Responses were received from 61 (64 percent). No additional effort was made to solicit information from non-responders.
After reviewing a sample of responses to the survey, we developed a taxonomy for classification of the findings. Eleven categories were identified and defined, which were intended to group findings with common characteristics of methodology or purpose (see categories). The process of defining categories was also informal, but provided a framework for identifying some common features of the findings submitted. An initial set of categories was proposed by one of the reviewers of the survey results, and this set was modified with input from two other project participants.
Once the list of categories was finalized, two physician/health services researchers reviewed the complete set of findings submitted by the PIs. Some PIs chose to list findings from a number of separate studies, while others reported a number of individual findings from a single study. The reviewers assigned findings for each discrete study to up to three different categories.
For the set of important findings submitted by COER-funded PI's, a total of 246 category assignments were made. Table 2 shows the frequency with which study findings were assigned to each category. The largest single category of important findings was "descriptive epidemiology": studies that described patterns of health care conditions or services. For example, one PI reported that the major independent risk factors for chlamydia infection are age less than 30, cervical friability, multiple sexual partners, and African American race. Another study reported the finding that the maximum estimated lost life expectancy for men with prostate cancer and a Gleason score of 5-7 was 4 to 5 years. "Practice variations," another primarily descriptive exercise, was the fifth largest category of findings. These studies generally identified unexplained variations in practice patterns that occurred in different regions, groups of providers, demographic subgroups, or time intervals. For example, one study identified a two-fold variation in the use of electrophysiology studies and implantable cardioverter-defibrillators by gender and race.
Another large group of studies identified as important compared the outcomes (clinical, quality of life, and/or economic) associated with different clinical interventions. Unlike the categories described above, these studies attempted to compare the outcomes associated with two or more alternative treatment strategies. In the majority of cases, these findings were derived from retrospective analysis of administrative databases. A smaller portion of these studies were prospective observational studies. Comparative effectiveness studies were able to associate various patterns of practice with differences in outcomes. Examples of "comparative effectiveness" findings include the observation that laparoscopic cholecystectomy is at least as safe as open cholecystectomy although the rate of common bile duct injuries remains uncertain. Another retrospective study confirmed the survival benefit from beta-blockers in patients after myocardial infarction and reported that this benefit is also observed for patients older than 75 years (who had not been included in the clinical trials).
Categories for findings reported by COER-funded primary investigators
Descriptive epidemiology: Natural history (follows patient without intervention), description of incidence or prevalence of condition, treatment, or complication, or identification of risk factors.
Comparative effectiveness: Comparison of outcomes associated with different treatments, diagnostic approaches, or other management strategies: "What works best."
Economic assessment: Compares costs and outcomes for different interventions, or looks at the costs of diseases or treatment without examining outcomes.
Patient-reported outcomes: Studies that report general or disease-specific quality of life, functional status, patient preferences, and/or patient satisfaction.
Practice variations: Comparison of rates of utilization of treatment, or service by some categorical group, such as region, demographic feature, provider, institution, time period.
Sociology of health care: Describes patient or provider knowledge, beliefs, and attitudes about conditions, treatments, process of care, etc.
Methodological development: Design, refinement, or validation of a technique for measuring, analyzing, or defining. Includes survey methods and risk adjustment/severity adjustment methods.
Modeling: Use of modeling techniques for decision analysis, predicting outcomes of interest, or determining cost-effectiveness of specific interventions.
Quality of care: Findings related to measurement of health care quality or provider performance.
Systematic review or meta-analysis: Summarizes existing literature with or without statistical analysis.
Legal, legislative, or regulatory: Describes or measures impact of an action of courts or government rulings that is related to health care.
Appendix 2. Abstract: Presented at SGIM Annual Meeting, April 1998, Chicago
The Outcome of Outcomes Research at the Agency for Health Care Policy and Research
Purpose: The purpose of this study is to assess the contribution of the Agency for Health Care Policy and Research (AHCPR) in supporting outcomes and effectiveness research (OER). Categorization of salient research findings identified by Principal Investigators (PIs) can be compared with the initial expectations of stakeholders so that future directions of OER can be determined.
Methods: A letter was sent from the Director of the Center for Outcomes and Effectiveness Research (COER) to all PIs who had received COER funding between 1989 and 1997, asking them to identify the "most salient findings" from each of their studies. They were asked to supply sufficient material for up to three slides. No specific criteria were provided to define "salience." A taxonomy of 11 categories was developed in order to group findings with common characteristics of methodology or purpose. Two health services researchers assigned findings to up to three categories for each discrete study.
Results: Responses were received from 61 (64 percent) of the 91 PIs, reporting on 115 studies. A total of 246 category assignments were made. Descriptive epidemiology, defined as a description of the incidence, prevalence, or natural history of a condition, treatment, or complication, was the most common type of study reported (24 percent of total assignments made). Studies of comparative effectiveness were also common (17 percent), in which patient outcomes were associated with different treatments, diagnostic approaches, or other management strategies. The majority of these were retrospective studies of administrative data. Other studies fit the remaining categories: economic assessments (12 percent), patient-reported outcomes (10 percent), practice variations (9 percent), sociology of health care (7 percent), methodological development (7 percent), modeling (7 percent), quality of care (4 percent), systematic reviews (2 percent), and policy-related (2 percent).
Conclusion: The studies most commonly identified as "salient" by COER-funded PIs were reports of existing patterns of care and outcomes, and their variability by location, provider, demographic subgroup, or time interval. PIs reported few studies that demonstrate clear superiority of one clinical strategy over another, successful incorporation of results into practice or policy, or interventions that had improved quality or lowered costs. The challenge for the next generation of outcomes research will be to build on descriptive studies and methodological advancements with the goal of measurably improving outcomes, quality, and efficiency of care.