The Outcome of Outcomes Research at AHCPR: Empirical Analysis
Empirical Analysis: Survey of COER-Funded Investigators
A survey was performed as part of this project in order to provide some systematic empirical information about the major findings that had been produced with Agency funding. The survey was mailed to all principal investigators (PIs) funded by COER between 1989 and 1997, asking them to describe their most important work (select for survey details of methods and results). The survey was sent to all recipients of COER funding, and was accompanied by a cover letter that asked PIs to "identify the most salient findings" from each of their outcomes/effectiveness research studies, and to provide bulleted material for up to three slides. The survey letter was mailed to 95 PIs. Responses were received from 61 (64 percent).
After reviewing a sample of responses to the survey, we developed a taxonomy for classification of the findings. Eleven categories were identified and defined, which were intended to group findings with common characteristics of methodology or purpose (select to access categories). Two reviewers assigned findings for each discrete study in up to three different categories.
For the set of important findings submitted by COER-funded PIs, a total of 246 category assignments were made. Table 2 shows the frequency with which study findings were assigned to each category. The largest single category of important findings was "descriptive epidemiology": studies that described patterns of health care conditions or services. For example, one PI reported that the major independent risk factors for chlamydia infection are age less than 30, cervical friability, multiple sexual partners, and African American race. Another study reported the finding that the maximum estimated lost life expectancy for men with prostate cancer and a Gleason score of 5-7 was 4 to 5 years. "Practice variations," another primarily descriptive exercise, was the fifth largest category of findings. These studies generally identified unexplained variations in practice patterns that occurred in different regions, groups of providers, demographic subgroups, or time intervals. For example, one study identified a two-fold variation in electrophysiology study and implantable cardioverter-defibrillator rates by gender and race. For reasons explained earlier, we consider all of these studies to be part of the spectrum of "outcomes research," even though some do not report outcomes.
Table 2: Classification of findings from COER-funded studies
|Sociology of health care||17||7|
|Quality of care||10||4|
|Systematic review or meta-analysis||5||2|
|Legal, legislative, or regulatory||4||2|
* Percentage does not add up to 100 because of rounding.
Another large group of studies identified by PIs as important compared the outcomes (clinical, quality of life, and/or economic) associated with different clinical interventions. Unlike the categories described above, these studies attempted to compare the outcomes associated with two or more alternative treatment strategies. In the majority of cases, these findings were derived from retrospective analysis of administrative databases. A smaller portion of these studies were prospective observational studies. Comparative effectiveness studies were able to associate various patterns of practice with differences in outcomes. Examples of "comparative effectiveness" findings include the observation that laparoscopic cholecystectomy is at least as safe as open cholecystectomy, although the rate of common bile duct injuries remains uncertain (Shea et al., 1996). Another retrospective study confirmed the survival benefit from beta blockers in patients after myocardial infarction, and reported that this benefit is also observed for patients older than 75 years (who were not included in the clinical trials) (Soumerai et al., 1997).
This survey provides information about the findings considered most significant by COER-funded investigators. One clear pattern that emerges is that many of these findings are descriptive in nature. In other words, these researchers have succeeded in developing strategies to describe who is affected by illness, how they are managed, and what the consequences of that management are. Often these descriptions were provided by creative use of data sources already in existence. In many cases, these descriptive efforts have produced unexpected results that lead to the identification of hypotheses for further studies (level 1 impacts).
The PIs also commonly reported studies that attempted to compare the effectiveness of alternative interventions. The methodological approach used most often was retrospective analysis of administrative data. As discussed later, inherent limitations of administrative data and retrospective analysis made it difficult to draw conclusions from these studies that would be persuasive for clinicians or policymakers. These studies generally called for additional studies to further explore the associations being reported, and sometimes observed that the situation was more complex than generally acknowledged.
Overall, the results of the PI survey are consistent with the fact that OER is a young field of research. Over the past 8 years, studies supported by COER have been most successful in providing descriptions of what actually occurs in health care, developing tools for measuring costs of care and patient-reported outcomes, and identifying topics for future research. These advances may appear disappointing when the measure of progress is clear, measurable impact, but the results are a prerequisite to implementing practice change or identifying effective interventions.
The survey results are also revealing for what they do not report. Few PIs reported findings that provide definitive information about the relative superiority of one treatment strategy over another. There are also few examples of findings that have been incorporated into policy (level 2 impacts) or clinical decisions (level 3), or interventions that have measurably improved quality or decreased costs of care (level 4). Much of what has been accomplished so far lays a solid foundation for future work. One of the main challenges for the next generation of outcomes studies is to move from description and methods development to problem solving and quality improvement.
Finally, the survey results also suggest a lack of alignment between what researchers and stakeholders consider important. Specifically, researchers did not appear to recognize the importance of dissemination, implementation, and impact. The first generation PORTs required a dissemination/evaluation component, but there was a substantial gap between that initial requirement and the execution. This was probably because this represented new territory for many of the researchers themselves, the challenges of carrying out the research itself, and limited enthusiasm about this aspect of the enterprise. Very little is known about the attitude of researchers toward this aspect of research, and whether it is changing over time. It will be important to understand these views, since it may be that they will interfere with strategic efforts to change the approach to and content of OER.
The Agency's ability to support research that translates to higher levels of impact depends on investigators who understand and value dissemination, implementation, and evaluation. In addition, it may be that some OER investigators are not the ideal vehicles for dissemination of their own research results. Just as architects hand off their blueprints to contractors when the time comes for implementing a design, it may be that other individuals or entities in health care will be more effective that the research community at translating the "blueprint" for care provided by OER into the programs and strategies needed to create change. As discussed in the options section, the Agency should consider a more active role in ensuring that the transfer of knowledge occurs more consistently.
Synthesis: Progress and Accomplishments
The field of OER has clearly progressed over the past decade, resulting in some fundamental changes in health care research, policy, and practice. Findings from the survey of PIs suggest some of the more substantial accomplishments. In addition, in-depth case studies were developed for specific research projects which appeared most influential and productive. Additional progress associated with the field was identified through prior critiques and the OER conference held in October 1997. This section describes some of the major categories of OER progress.
It is worth noting that many of the accomplishments identified below would be categorized in our conceptual framework as level 1 impacts. In several examples described below, the OER findings initiated some action by a health care "change agent," i.e., a level 2 impact. The fact that a medical professional society or health delivery system commits time, staff, and resources to pursuing OER findings has two important implications. First, it reflects the fact that these organizations are adequately convinced of the practical importance of the research. Second, it is through their efforts that level 3 and 4 impacts are likely to occur.
AHCPR has been a leading U. S. organization in several major conceptual shifts in health care research and practice. At least three conceptual developments have been strongly influenced by AHCPR-sponsored work:
- The increasing recognition that evidence, rather than opinion, should guide clinical decisionmaking.
- The acceptance that a broader range of patient outcomes needs to be measured in order to understand the true benefits and risks of health care interventions.
- The perspective that research priorities should be guided in part by public health needs.
Evidence should guide clinical decisions
Early studies in geographic variation and appropriateness of care were responsible for documenting the inadequate scientific basis for medical decisionmaking in virtually any clinical discipline that was explored (Wennberg, 1984). The paucity of reliable evidence has now become well accepted. These studies enabled the medical and policy community to acknowledge that the "doctor didn't always know best," and that there were more formal and reliable techniques to determine effectiveness. AHCPR has been the main venue of support for this intellectual approach.
There is now widespread acceptance of the view that scientific information should be systematically applied to health care decisionmaking. As one reflection of this evolution, dozens of medical professional organizations have taken on the task of developing evidence-based guidelines for their membership (ECRI, 1998). A number of these groups have also become active collaborators in OER studies. The American Academy of Ophthalmology and the American Urologic Association were critical collaborators in the cataract and Prostate Disease PORTs, respectively. The AMA recently found that over 50 percent of its subspecialty societies are now engaged in activities related to outcomes research.
Even more fundamentally, clinicians are now exposed early in their training to the idea that the opinion of experts and accumulated personal experience are not the most reliable sources of input for clinical decisions. It is difficult both to quantify these changes in attitudes and expectations, and to estimate what fraction of the changes is truly attributable to OER. However, it is reasonable to argue that the increased emphasis on applying published literature to clinical decisionmaking will increase the use of effective interventions and improve health outcomes. It is also legitimate to observe that AHCPR has taken a leading position in the effort to increase the scientific basis of health care decisionmaking, and that OER investigators have facilitated the relatively rapid adoption of this perspective.
Outcomes that people experience—and care about
The second conceptual shift contributed to by AHCPR is the recognition that determination of the value of health care services requires an expansion of metrics beyond the traditional measures of mortality and major morbidity. Many health care interventions significantly affect patient function and well-being while having little measurable impact on risk of death or disability. What matters for cataract patients is not their ability to read a Snellen eye chart, but the degree to which their visual problems affect their usual activities. Patients with prostate hypertrophy care less about stream force that they do about the number of trips to the bathroom during the night. AHCPR was instrumental in supporting and promoting this fundamental shift in the approach to measuring risks and benefits of health care interventions.
OER investigators then developed and validated the measurement tools required to conduct this research (see Table 2). As one example, the VF-14 was developed to measure how well patients with cataract disease were able to pursue their usual activities (Steinberg et al., 1994). The measure is now required in all clinical trials sponsored by the National Eye Institute testing the benefits of new technologies and procedures for cataract patients. Many insurance companies (including Medicare) require that the VF-14 be reported as a condition of claims payment, and it has been adapted for use in at least a dozen other countries (E. Steinberg, personal communication, April 1998). This widespread adoption of the VF-14, also observed for a number of other tools developed with AHCPR support, indicates that decisionmakers believe such tools improve the quality and appropriateness of care.
As part of this shift to focus on outcomes that matter to patients, the concept of shared decisionmaking has also taken hold. Once it was more widely accepted that the value of many health care interventions could only be determined by understanding and measuring the preferences and values of individual patients, the importance of including those factors in clinical decisions became obvious. OER investigators have developed methods for informing patients about the likelihood of various outcomes from their treatment, and helping solicit their judgments about the importance of those outcomes. This process has been shown to influence decisionmaking for patients with back pain, prostate disease, breast cancer, and other conditions. Patients who participate in decisionmaking have been shown to be more satisfied and have better physical function than patients who do not, even though they may make the same decisions about care (Barry et al., 1997).
Asking the right questions
AHCPR has provided a model of priority-setting for research that attempts to be responsive to actual health needs of the population. The development of scientific knowledge in the biomedical sciences in the U.S. has been driven largely by the concept of "scientific opportunity" rather than a strategic evaluation of population health needs. AHCPR's comparatively modest budget inspired a targeted approach, using an explicit process of looking at such factors as prevalence, burden of illness, cost, etc., in order to define the domains of research investigation (Institute of Medicine, 1990). As part of this process, AHCPR defined potential research areas in ways that would be most clinically meaningful, such as targeting broad clinical conditions (e.g., community-acquired pneumonia) or common clinical presentations (e.g., low back pain). Another unique element of this priority-setting process was the use of initial studies which examined patterns of actual practice.
The growth of clinical evaluation sciences (of which OER is a part) with the development of a priority-setting process that is driven by public health need, has been prompted by an understanding that OER serves a practical purpose, in that the ultimate goal is to bring scientific clarity to applied decisionmaking and policymaking at all levels of the health care system. Research efforts must therefore be informed and directed by a detailed appreciation of the magnitude and substance of those problems, as defined by patients, providers, and other stakeholders. The relationship between the biomedical sciences, health care delivery and financing, medical innovation, and clinical evaluation sciences invites further clarification—an important future challenge for COER/AHCPR. Biomedical research serves the role of generating and testing the medical innovations and breakthroughs. Understanding the real value of these innovations, and how best to deploy them, is the role of OER and the other clinical evaluation sciences.
Challenging Clinical Dogma
The body of OER studies represents an aerial view of clinical practice patterns at a scale that had previously not been available. These OER studies have often provided descriptive data that challenged prevailing clinical ideas about how best to manage specific clinical problems. By documenting patterns of care or outcomes that are inconsistent with existing understandings of disease processes and management, a tension is created that can lead to further study or directly to changes in practice.
For example, a study of otitis media in a pediatric Medicaid population showed that children with otitis media were more likely to receive a second course of antibiotics if they were initially given an expensive name-brand product when compared with a generic product (Berman et al., 1997). This finding is counter-intuitive, and helps to raise questions about one of the specific explanations given for using more recently developed drug therapies (i.e., that failure rates are higher for generic antibiotics). This study result does not conclusively prove that cure rates are equal for the two groups of drugs, since the same pattern of prescriptions would be observed if, for example, the clinicians who use expensive drugs are also more likely to prescribe a second course of therapy. The primary value of this study is confirmation in a large population (over 12,000 cases) that there is no evidence of worse outcomes with generic, narrow-spectrum antibiotics.
"Common clinical wisdom" was also challenged by the classic BPH studies performed by Wennberg, Barry, and others. Their research revealed that the rate of impotence and incontinence after surgery for BPH was much higher than most surgeons and patients believed. Furthermore, they determined that patients had a high range of tolerance for BPH symptoms (Barry et al., 1995). These two observations have important implications for a patient considering BPH surgery, given that personal decisions about the tradeoff between one set of symptoms and another are necessary. Other work by the same team showed that mortality was not increased when BPH surgery was performed on older patients (Barry et al., 1988). Many surgeons recommended early surgery for BPH, based on the belief that younger patients would tolerate the procedure better, that they would need the surgery eventually, and that complication rates were reasonably low. In this case, descriptive information from observational studies challenged several accepted concepts about the outcomes of care. The BPH studies are credited with having contributed to a recent trend toward less aggressive therapy for prostate symptoms.
Development of research methods and tools
The most concrete accomplishments of the decade of OER are the tools and analytic methods that have been developed. These include strategies for conducting systematic reviews and meta-analysis, instruments for measuring health outcomes important to patients, and sophisticated techniques for analyzing observational data to adjust for disease severity and minimize bias.
Decision analysis, meta-analysis, use of claims data, cost-effectiveness analysis, assessment of patient preferences and utilities, methods to adjust for patient comorbidity and severity of illness, and techniques to adjust for selection bias in observational studies are all methodological areas that did not begin with AHCPR's investment in OER but benefited substantially from the cross-project work teams. An important challenge will be to further articulate the value of these methodological advances, despite the fact that multiple private sector entities engaged in OER have identified their support as an unique public good.
The linkage between these technical accomplishments and actual improvements in practice and outcomes is not generally apparent. One such illustration is provided by the Northern New England Cardiovascular Disease Study Group (NNECDSG). This voluntary group of cardiac surgical practices used methods of data collection and risk adjustment developed by the Ischemic Heart Disease PORT team in order to establish a registry of patients who underwent coronary revascularization. There are now close to 100,000 procedures in this database. Most cardiovascular surgeons in northern New England provide data on all procedures. Use of this database has been associated with a significant reduction in mortality rates for participating surgeons. This work capitalized on knowledge that variation in mortality rates existed, and that measurement and exploration of the sources of that variation could lead to improvements in care. Surgeons from this network pay to participate, complete all data forms themselves, and travel to each other's hospitals to learn techniques from the best performers. This work was built on the conceptual contributions of OER and benefited from the analytic tools and data collection methods developed with AHCPR support.
Another indication of the value of technical methodological advances to the health care industry is the increased demands for information on both cost and value. This demand has created an important impetus for additional methodological development in the area of cost-effectiveness analysis.
Identifying knowledge gaps and prompting needed studies
Some findings from OER may not be definitive enough to influence practice, but may help direct future research efforts. For example, The cataract PORT team used a large Medicare database supplemented with primary data collection to make a number of novel observations about cataract surgery. One study reported that the risk of retinal detachment is increased in patients who underwent YAG laser capsulotomy (Javitt et al., 1992). A number of methodological limitations reduce the reliability of this conclusion, but the importance of the finding was that it raised the possibility of a causal relationship that was not previously suspected.
Having identified areas of uncertainty in the science base for their practice, the American Academy of Ophthalmology (AAO) teamed up with the PORT team to develop strategies for gathering better data. The result of this collaboration was the establishment of a network of ophthalmologists who are reporting procedural data and outcomes on their patients (now called the National Eyecare Outcomes Network, or NEON). This database, designed for research and quality improvement purposes, will be capable of answering important remaining questions about the outcomes of cataract surgery. The database is now 10 times larger that the original study conducted by the PORT. The AAO has invested over $1 million in this effort, and individual physicians pay a fee to participate. The AAO and the physicians believe that this database enables them to provide reliable performance data on their cataract surgery when competing for managed care contracts. They also use the data to detect differences in patient outcomes that may allow specific providers or groups to identify quality problems and improve quality of care.
Another category of studies that were supported in significant numbers by AHCPR's COER was a group of analyses that documented unexplained differences in treatment patterns and health outcomes, particularly by gender and by race. Numerous studies in a variety of clinical areas revealed higher utilization of services and greater morbidity and mortality, with white males usually using most services and experiencing better outcomes. Very few such studies had been carried out prior to the 1980s. While these studies are not able to explain the specific reasons for the differences observed, they have increased attention to this problem among researchers and policymakers. One direct impact of this work is the recognition that clinical trials and other research studies often include relatively few women and minorities, therefore much less is known about appropriate standards of care for these population. Research funding agencies now require that investigators include in their proposals an explicit plan for including adequate numbers of women and minorities. The HHS initiative to eliminate racial disparities in health outcomes by 2010 is another consequence of these studies.
Developing the manpower for outcomes research
Public funding for OER has produced a network of institutions and trained investigators capable of carrying out OER. A relatively small core of academic centers and professionals has grown to include a large number of individuals and institutions pursuing outcomes studies in the public and private sectors. Divisions, programs, centers, and other organizational entities dedicated to outcomes studies are being established by hospitals, managed care organizations (MCOs), specialty societies, and others. Most major pharmaceutical companies have established internal groups of outcomes specialists. An IOM report on health services research training stated clearly that there is a (perceived) "...shortage of outcomes researchers" (Careers in clinical research: obstacles and opportunities. Committee on Addressing Career Paths for Clinical Research, Institute of Medicine. William Kelley and Mark Randolph, eds. National Academy Press, Washington DC, 1994.) The Lewin report of private-sector OER also confirmed the central importance of AHCPR support for training the majority of leaders in the field today (Lewin Group, Outcomes and Effectiveness Research in the Private Sector: Final Report [Fairfax, VA: Lewin Group, 1997]).