The Outcome of Outcomes Research at AHCPR: Recommendations
Outcomes research, like technology assessment, can be thought of as a set of analytic methods that produce information about the benefits, risks, and costs of health care services. Outcomes studies can support a particular decision, but they do not make decisions, and therefore have no impact unless they are acted upon. The impact of this enterprise therefore is critically dependent on the action of decisionmakers in health care, who must be aware of, understand, accept, and use this information. The options described below are each influenced by the objective of enhancing OER impact, taking into consideration this perspective on the essential nature of OER as knowledge, not action.
Enhancing the impact of OER in the dissemination phase has often been thought of as the final activity of a research effort. Dissemination is listed as the last activity of the PORT projects, and generally received the least attention and achieved the least success. Educational outreach, and/or social marketing, represent the standard of success in this aspect of most OER studies. In many cases, dissemination activities simply received little serious attention. In addition, researchers' goals for dissemination (publications; dissemination of knowledge) and decisionmakers' needs (positive changes in practice) are not necessarily linked.
To affect practice and outcomes, these issues must be considered much earlier in the research process, and should influence how research questions are framed, what methods used, etc. It would not be premature to consider issues of dissemination as part of the criteria that enter into the funding decision—do the questions addressed and the proposed approach have a reasonable chance of affecting policy and practice? Critical components of successful marketing include a market analysis, which helps to determine which products are desired by consumers, and what features those products should have. For consumers of OER, the health care decisionmakers, very little effective effort has gone into linking research funding to information needs.
Several general options to enhance the probability of positive impact are identified and discussed in greater detail in this section of the report:
- Prioritize research according to some measure of "likelihood of impact," with particular preference given to proposed studies that build on previously funded work.
- Fund more "problem-solving" studies that explicitly look at various strategies and interventions to change practice.
- Promote improvements in strategies and methods used to study effectiveness.
- Promote development and use of practical patient-oriented outcome measures.
- Identify research findings that should be implemented, and provide usable information and technical assistance to "change agents" that are in a position to influence practice.
- Create a surveillance mechanism that will systematically document changes in practice that may be related to Agency-supported work.
Set Research Priorities: Orchestrated Follow-On
One of the most significant contributions of AHCPR and COER in the past decade has been to identify specific gaps in scientific knowledge that lead to high levels of practice variation and suboptimal outcomes. Many studies supported by COER over the years have served as hypothesis-generating studies, but there has not been a systematic effort to orchestrate the hypothesis-testing follow-on. COER could take on the responsibility to make sure that once these critical knowledge gaps are identified, they are addressed in follow-on studies. Limited resources prevent the Agency from funding all such studies, but there are increasing opportunities to leverage outside resources. Private-sector organizations and other public Agencies have considerable resources and increasing motivation to undertake applied research. COER should take on the role of a "catalyst" for additional research, attempting to leverage these resources when an appropriate topic has been identified.
The goal of increasing measurable impact can be approached by selecting research topics and methods that optimize likely impact. To the extent that COER is able to link internally supported hypothesis-generating studies with more applied studies, the Center will be creating a chain of efforts that will ultimately lead to measurable improvements in patient care. Proactive efforts to forge these linkages will create a track record of accomplishments that link the descriptive studies (e.g., level 1) to visible impacts on policies, practices, and outcomes (levels 2 through 4).
For example, the study of treatment of otitis media in the Medicaid population was described earlier. The principal investigator for those studies went on to collaborate with policymaking organizations to develop guidelines for antibiotic choice. However, despite the clear suggestions made by this study that clinical practice should change, we were not able to identify any followup reports of specific efforts to change practice in the direction indicated.
There are many other examples of OER studies that provide descriptive findings that strongly suggest ways in which care and outcomes could be improved. There are, however, only rare—and serendipitous—examples of those finding being effectively pursued to document that the proposed changes in practice can be achieved, and that they result in improved outcomes. Currently, it is left up to investigators to pursue the logical next steps from their research, and they may lack the interest, knowledge, or resources to do so. There is no strategic plan that would give priority to this follow-on work.
There is clearly a tension between efforts by the Agency to channel research in desired direction, and the need to preserve investigator initiative and creativity in research. While unchanneled intellectual exploration is a time-tested and vital approach to research, it may not be entirely satisfactory for AHCPR, given the expectations for measurable impact discussed throughout this report. Similarly, there is an implicit tension associated with the strategy of giving preference to promising follow-on work over supporting work in a new area. A specific approach would be to develop a targeted initiative promoting follow-on efforts for completed studies, coupled with supplements to studies in progress intended to enhance impact.
Another practical aspect of strategically pursuing research opportunities through higher levels of impact is to identify potential partners. The Health Care Financing Administration (HCFA) and other payers, MCOs and medical groups, medical professional organizations, and medical product manufacturers would all be potential research partners. Collaboration with these organizations would not only allow AHCPR to leverage resources, but would ensure that potential studies are crafted to meet the applied needs of these organizations. If this approach is successful, dissemination and implementation are built into the research plan, since the target of impact is also a partner in study design and funding. AHCPR could produce a Request for Applications (RFA) that motivates the investigators to develop partnerships by requiring that the project obtain 50 percent of its funding from a source other than AHCPR. This would serve both to leverage AHCPR resources, and would also promote partnerships with "end-user" organizations to design and conduct research. Effective and efficient strategies for establishing these partnerships need further development.
Clarify Criteria for Relevance and Importance
The review process for new grants will need to adopt new criteria that give preference to studies that build upon basic findings and convert those findings into system impacts. This "potential impact review" would be added before or after the existing scientific review, and would specifically consider the likely impact of research from a clinical and policy perspective. It may be valuable to have health care decisionmakers with operational responsibilities participate in this "potential impact" review process, rather than solely the research community, either prior to funding or as part of a retrospective review.
Review criteria should also be sensitive to the likelihood that private sector OER will be adequate. The overview of private sector OER recently completed by Lewin provides a description of the types of OER performed in the private sector, and what gaps remain in these activities (Mendelsohn et al., 1998). The main conclusions from the Lewin report are that the OER of private-sector organizations are closely tied to the financial interests and business viability of the respective sponsors. They are most likely to take on high-prevalence conditions, with high unit costs, but also those for which a potentially high-cost new technology, service, or product is available. Important but commercially unattractive problems, such as sexually transmitted diseases (STDs) and substance abuse, are less commonly addressed. Private-sector organizations are less likely to invest in the development of tools, methods, or basic research strategies. Most health-related quality of life (HRQL) instruments and analytic methods that are well designed and validated and used by private organizations were developed with public sector support.
While the Agency should consider increasing its funding for more "applied" studies, there is a need to determine how best to balance the research portfolio. The tradeoff to explore is between studies that demonstrate near-term high-level impacts, versus more basic studies (tools development, hypothesis generation) for which there may be important long-term payoff. AHCPR should invest in studies that provide level 1 impacts, while also supporting an adequate number of studies that target level 3 and level 4 impacts. One model for this "pyramid of effort" (or funding) for OER would be wide at the bottom (develop methods, identify important problems) and narrow at the top (implement programs to change practice). This is depicted in Figure 1 (12 KB).
Figure 1 primarily reflects the gradations for types of study that are likely only to be supported by the public sector, and types of studies that can potentially be supported in the private sector. A focus on "special populations," including children, minorities, and women represents a unique public interest.
Make OER Approach Responsive to Existing Quality of the Evidence
The existing quality of research completed in a clinical field has implications for what level of impact is likely over a short time period as a result of OER. In addition, the current state of evidence will also have implications for what type of OER is most likely to be valuable. For example, in fields such as surgery, for which the state of the science begins at a low level, it may be most efficient to focus on studies that raise questions, or identify important hypotheses. In addition, development of patient-reported outcome tools, methods for severity adjustment, etc., may be needed before more sophisticated studies can be done.
It should be possible to develop an approach for assessing the existing quality of evidence in a given field. This would be parallel to a "readiness to change" assessment used for behavioral interventions. To improve impact, OER studies should be supported in fields that are determined to be "ready" for change. Again, it will be necessary to maintain a balanced portfolio, as more basic studies will still be needed in clinical disciplines for which the evidence base is less well developed.
Increase Emphasis on Problem-Solving
The first decade of OER has been particularly successful in describing what actually occurs in medical practice, and in creating and refining tools to be used in studying what works. It has, however, become clear that these activities themselves do not lead inexorably to practice improvement. A critical question for COER is to determine the appropriate fraction of effort and resources that should be devoted to studies that incorporate strategies for change as part of the study. This next stage of OER would focus attention on creating designs and blueprints for improving care, and selectively applying the OER tools when they are needed. It would represent a move from a "tool-using" culture to a "problem-solving" culture, for which the primary objective is developing an empirical science of clinical improvement. The COER could be refashioned into the "CHOIR" (the Center for Health Outcomes Improvement Research).
In the past, an OER investigator might begin with a clinical condition and a particular large dataset and ask the question: "What can I learn about condition x from database y using method z?" The CHOIR approach to this would frame the question as: "What do decisionmakers need to know about condition x, and which method and database are most likely to provide this knowledge?"
This change would require a new emphasis on a number of activities at AHCPR. First, a systematic effort would be needed to define in detail what high priority problems remain for specific clinical conditions. This will require more than identifying conditions that are high cost, or associated with large variations in practice. More specific, testable hypotheses about particular clinical practices need to be defined. These could be derived in part from intramural analysis of claims and other data.
The most important step is to make sure to ask the right question and address that question with a methodology that is appropriate. There should be a "scientific assessment of the evidence needed to stimulate practice change" (Clancy and Eisenberg, 1998). Over time, it may be possible to develop standardized approaches to deciding what type of study is needed, given specific features of the issue being addressed. One model for this approach is a recent analysis of the value of trials vs. observational studies to support policy decisions (Hornberger and Wrone, 1997). A major consideration in choosing an analytic approach will be to ask who the decisionmakers are that the research is intended to influence, and what sort of evidence will be required to alter their decisions. These questions merit serious empirical analysis.
This process can go on internally at AHCPR, and should also be required of investigators submitting proposals to the Agency. In either case, the shift in culture should ensure that future studies are explicitly linked to problems in need of solutions, and use study designs that reflect the specific features of the problem being addressed. Because this approach represents a significant shift for AHCPR, it may be most practical to solicit proposals for prototype projects. Development of the RFA would itself allow focused internal discussion about the conceptual, methodological, and operational challenges of adopting this strategic approach.
Promote Improvements in Strategies to Study Effectiveness
One of the specific areas in which COER work has been less successful than hoped is in providing definitive answers about the comparative effectiveness and cost-effectiveness of health care interventions (i.e, what works best in health care). We need to develop innovative ways to address the many unanswered questions about effectiveness and cost-effectiveness, including the effect of environmental characteristics.
Little progress has been made in addressing the well-recognized limitations of prospective clinical trials (i.e., cost, generalizability, timeliness). While the Agency has increased its support for RCTs, these still tend to resemble traditional efficacy studies, and therefore remain complex, slow, and costly. There has been no concerted effort to craft a new methodological, organizational, and ethical framework for these studies. The "conceptual infrastructure" for conducting effectiveness trials remains to be developed.
Effectiveness trials attempt to blend the "real world" context of database studies with the bias protections of traditional RCTs. They have broad eligibility criteria, non-intrusive study protocols, and make efficient use of data collected as part of routine practice. These studies would seek to address issues of effectiveness in the context of the expanding diversity of health care providers, settings, and financing—looking at the outcomes associated with, for example, long-term care facilities, day surgery, various models of managed care, paraprofessionals, providers of alternative medicine, etc.
Essentially, this approach minimizes the distinction between research and clinical practice, for the purpose of creating a systematic and efficient mechanism that allows the medical profession to learn while doing. By requiring a linkage between research and practice, effectiveness trials are by themselves one approach to effectively translating evidence into practice. Personal involvement and participation is one of the important conditions for producing changes in professional behavior (Davis et al., 1994). Informal evidence of this is the dramatic increase in the use of aspirin after MI in England followed the ISIS trial (Collins and Julian, 1991), which may well have resulted in part because a critical mass of British physicians were investigators in that large trial. Practice did not change among Italian physicians until the ISIS results were replicated in Italy in the GISSI trial.
COER could engage in a major initiative to create and refine new approaches to conducting prospective effectiveness trials. This effort could be similar in visibility and scope to the "effectiveness initiative" of the late 1980s, which motivated the research community to develop strategies for using large administrative databases as an alternative to traditional clinical trials (Roper et al., 1988). The creative energy and ingenuity of the health services research community was directed at this problem with notable success, and it is likely that similar contributions could be made in crafting new approaches to studying effectiveness in real-world settings using prospective, experimental methods. Current practice research networks represent one important opportunity for establishing this initiative and the requisite infrastructure.
This recommendation would require a significant amount of energy and leadership from the Agency to be pursued successfully. It may be a similar undertaking to that required to launch the RCT as the "gold standard" of efficacy research. AHCPR has already accomplished something of similar magnitude with the PORTs, a large-scale effort that similarly created a new model of conducting research. As with the PORTs, the new effectiveness initiative would support methods development, enhance specific technical skill in the research community, promote conceptual discussion of effectiveness trials, and create networks of researchers and institutions. In the same way that clinical guidelines development and observational studies were given a major boost in the early 90s, this new initiative would support a subpopulation of researchers to identify and explore all of these issues, and hopefully launch this field as successfully as OER was launched.
Promoting creative strategies for prospective studies would require a significant conceptual shift. Prospective effectiveness studies can only be undertaken when professionals involved in delivering care become participants in the research enterprise. These studies can best be done as partnerships between researchers, delivery systems, and funding organizations. The operational and organizational challenges of this are quite significant. For example, creating the infrastructure and techniques which facilitate data collection in routine practice and implementing them in ways that do not increase clinician workload will obviously require some ingenuity. Similarly, issues of identifying how funding responsibilities should be shared, obtaining adequate sample size, managing conflicts of interest among partners, recruiting patients, obtaining patient consent, and protecting patient confidentiality all need considerable attention.
Analysis of large administrative databases and observational studies of all kinds should continue to be an activity supported by COER. A shift in the weight of emphasis toward prospective studies for some period of time will help to establish the feasibility and cost of developing and implementing such methods. It may be useful to have an initial conference bringing together investigators who have already worked in this area, and who have at least identified many of the complex challenges. This event could also serve as an announcement of this initiative by the Agency. Following this, an RFA could be issued that would support several prototype effectiveness trials.
Promote Routine Use of Outcomes Measures
Routine use of patient-reported outcome measures in clinical practice remains the exception (e.g., the prostate symptom score developed by the BPH PORT and the American Urological Association [AUA]) rather than the rule. Expanding the metrics for answering the question "What works?" cannot be successful until clinicians incorporate these tools in daily practice. A strategic focus on clarifying the validity of these measures and identifying strategies that encourage their use is essential to COER's future efforts. Additional work on the incorporation into practice, feasibility, and credibility of disease-specific and general outcomes measures, and development of tools and strategies to make this possible (e.g., a database of measures analogous to CONQUEST) are a critical requisite to getting to outcomes that people experience and care about. HCFA's "Health Outcomes Survey" measure, and other measures being developed by the Measures Advisory Panels (MAPs) at NCQA, are likely to provide a new opportunity for pushing the current frontier, but important conceptual and philosophical challenges must first be addressed.
For example, because current efforts to include patient-reported outcome measures are conceived as mailed surveys, the feedback loop to clinicians is assumed rather than specified. The importance of incorporating patients' preferences and values is widely acknowledged, but practical approaches for valid and reliable measurement remain to be determined. Work that clarifies the conceptual domains of HRQL measures and permits enhanced interpretability across measures is needed before many measures can be widely used (Gill et al., 1994). Strategies for estimating the clinical significance of differences in outcomes remain much less well developed than those for statistical significance (Deyo and Patrick, 1995). One difficulty in promoting widespread use of patient-reported outcomes measures is that clinicians are unlikely to use them routinely unless the reports are as familiar to them as blood pressure and other physiologic measures—but this cannot occur until outcomes measures are developed that are easy to include in day-to-day practice. Improvements in outcomes measures and development of strategies to encourage their routine use are an essential future direction for COER.
Continue Tools and Methods Development
New challenges in terms of methods development and refinement have developed from the previously discussed advances in OER as well as the shift towards projects that will have greater impact on policies, practices, and outcomes. One such challenge is in the developing area of cost-effectiveness analysis (CEA). Contemporary medicine has been greatly influenced by increases in health care costs to over 14 percent of our Nation's gross domestic product with discussion of rationing, the shift towards managed care, involvement of consumers in decisionmaking, and chronic diseases in which there are often tradeoffs between quantity and quality of life. Such changes have resulted in greater demands for information about value. While the use of CEA has increased dramatically in recent years (Elixhauser et al., 1998), there is little agreement about methodologies in determining both costs and metrics of effectiveness making it difficult to assess the validity of a study or comparisons between studies. As CEA examines incremental change in resources required and health effects gained by using one intervention compared with another, some studies base comparisons on the best available alternative, while others are based on viable low-cost or do-nothing alternatives. Other unresolved areas include which costs to include and how to measure them, how to discount costs and outcomes over time and whose preferences to use and how to measure them in determining effectiveness.
The first decade of OER has demonstrated that database analysis and systematic literature review cannot substitute for clinical trials. Instead, it has raised the question of how trials can best be designed so that results are rapidly applied to practice. Although the randomized controlled trial is presently considered the gold standard, other designs including minimization have been suggested as alternatives. Under minimization, group allocation is not based solely on chance (Treasure and MacRae, 1998). Instead, patients are stratified by known or suspected determinants of outcome, such as diabetes and hypertension in the study of coronary artery disease. Such stratification substitutes for multivariate analysis in other study designs. With further experience with and understanding of trial designs, researchers should be able to more appropriately choose methodologies to match the question being asked and the level of evidence needed to stimulate change by practitioners and policymakers.
Create the Capacity to Build Upon and Transfer Knowledge
The Agency should consider developing the capacity to identify important research findings (generally level 1 impacts), and to assist those in moving to higher levels. This capacity should be proactive, intensive and consistent, and not left solely to the ingenuity of each research team. The research methods (and probably the resources) for confirmatory and follow-on studies suggested by OER work require technical skills and organizational knowledge that is not well developed within the mainstream health services research community. In addition, the skills to disseminate information effectively and provide support to health care organizations that hope to translate research into practice are also not common. Academic incentives tend to favor analytic exercises over dissemination, since the former require less effort to produce publishable results.
A dedicated knowledge transfer function would systematically consider findings from OER studies and carefully assess them to determine how best that work can be translated into higher levels of impact. In some cases, the appropriate next step would be a specific type of follow-on study, while in other cases, some effort to influence practice in the direction suggested by the study would be indicated. Under other conditions, it may be appropriate to convene affected specialists to discuss the clinical and research implications of the findings, and have this group suggest a follow-on strategy. The core theme to each of these alternatives is that the Agency should participate in developing a structured, scientific, and systematic approach for guiding research from its basic stages through clinical applications, and finally to measured improvements in outcome.
Knowledge transfer could be accomplished by working with individual investigators, creating an AHCPR center devoted to knowledge transfer, or creating an external entity with this function. The Center for OER Knowledge Transfer could be supported by the Agency, possibly through a relationship similar to that between the Agency and Evidence-based Practice Centers. An internal or external center would develop a strategy for collecting and documenting examples of OER impact stories in real time. In addition, the Center could serve as the coordinator to ensure that research is acted upon. Center staff would act as brokers for new, useful knowledge—working out how particular knowledge might best be acted upon, making necessary connections between user groups, new knowledge, and potential funding. The Center could then also provide technical assistance to organizations in implementation projects.
Promote Clear Vision of Roles and Impact
For COER to be seen as accomplishing its goals, those goals must be agreed upon and clearly stated. A revised and updated Program Announcement, coupled with articles in the peer-reviewed literature, should be developed to articulate and reinforce these key themes. These themes should be documented, widely distributed, and frequently referred to in Center activities, deliberations, and presentations. If, for example, COER decides that the largest part of its activities should be devoted to the development of basic tools and methods, then this strategy should be clearly described and explained. One benefit of explicit and measurable goals for COER is that it will be possible to review efforts of the Center and evaluate the degree to which its stated objectives have been accomplished. It should also be possible to determine whether the assumptions and beliefs upon which these objectives were established were valid. One of the difficulties of the current efforts to learn from past experiences is the lack of an explicit framework and stated objectives for the work that has been performed.
For example, COER might decide to emphasize basic studies to develop and validate research tools and outcome measurement instruments. For this approach to be considered successful, the output of these efforts would have to be incorporated by other researchers and by health care organizations. The Agency could set explicit goals concerning the adoption of these tools and methods. At the same time, surveillance systems would be introduced to detect and measure these effects. In 3 to 5 years, the actual accomplishments could be compared to the stated objectives, and reasons for success or failure could be explored.
Clarifying the vision for OER requires that important terminology and concepts are clear to those who conduct OER, and those who are expected to use it. Responses to calls for priority topics from the Agency suggest that there is persistent confusion about the difference between OER and guidelines. Some attention has been given to clarifying the relationship between outcomes studies, effectiveness studies, outcomes management, quality measurement, etc. The quality-accountability continuum in use at AHCPR for the past several years provides some conceptual linkage for these related activities. Translating that continuum into an internal strategy for channeling resources would be a useful planning exercise.
It is also important to clearly convey the complementary nature of OER studies with traditional basic science and clinical research. Policymakers need to have a firm understanding that OER provides the link between basic research and clinical practice. While clinical research identifies potentially valuable new drugs and procedures, OER studies help to clarify how new technologies can be applied in practice with greatest health benefit and efficiency. For example, in the case of congestive heart failure (CHF), clinical trials have demonstrated the efficacy of angiotensin-converting enzyme (ACE) inhibitors and other therapies. It is not, however, within the domain of traditional clinical research to examine such questions as which CHF patients benefit from being managed in an intensive care unit (ICU) setting. AHCPR has sponsored studies that show that many low-risk patients are managed in the ICU, and some patients are inadequately managed outside the critical care setting. OER investigation into these issues helps to generate strategies to most effectively and efficiently deploy clinical interventions and systems of care. Although these studies complement clinical research advances in managing CHF, the value of OER is currently less well appreciated by congressional funders than the value of clinical research.
Increase Interaction with Major Stakeholders and Customers
The prevailing model of health care delivery has changed dramatically, and the science of OER must be responsive to these changes. As an applied discipline, OER is faced with a constantly shifting target. A high level of interaction with stakeholders in the health care system will ensure that the basic studies are supportive of real problems faced by those involved in health care delivery. The adoption and use of information by the private sector will serve to increase the apparent and measurable impact of more basic studies supported by the Agency. While health care providers and other private sector groups may have incentives and resources to conduct applied research or demonstration programs, they may not have all of the technical resources or perspective necessary to initiate these studies. COER (or CHOIR) can serve as a catalyst to engage the resources and incentives in the private sector under these circumstances. This again requires a higher level of engagement and interaction with these organizations than there has been in the past.
COER should continue to develop a detailed understanding of the operational organizations that comprise the health care system. Ideally, MCO managers and medical directors, hospital administrators, insurance company business managers, medical product industry marketing directors, pharmaceutical product managers, private sector outcomes researchers, and others could directly participate in a range of Center activities. These individuals could serve on grant review committees, participate in panels and workshops, be assembled in regular focus groups, be involved in site visits, etc.
The level of involvement with other organizations interested in conducting or using outcomes studies could also be increased. Medical professional organizations, medical group practices, and others bring an understanding of the needs and challenges of their respective members and institutions. Through a greater level of ongoing interaction, these needs could become factors that influenced decisions about research priorities and methodology. Cooperation could also result in concrete opportunities to collaborate on specific studies, bringing these organizations into a more active research role, and ensuring that their real-world perspectives were integrated into these research efforts.
It may also be useful to conduct a major survey or convene focus groups of OER stakeholders (managed care organizations, pharmaceuticals, consumers, others) to determine opportunities and strategies for better serving end-users. However, the direct involvement of these individuals in the activities of AHCPR will increase the degree to which their perspectives are represented.
The effectiveness initiative, launched a decade ago, has yielded both important findings and opportunities for improvement. A new emphasis on evidence-based decisionmaking and interest in assessing and improving quality of care, along with interest among a broad array of private-sector organizations in OER, set the stage for this analysis. The attempt to systematically review research output from an organization has few clear precedents, and necessarily reflects the limited perspective of the reviewers themselves, as well as the difficulty of determining the value of generated knowledge. A key conclusion from this analysis is that the accomplishments of the first decade provide a foundations from which to advance. However, it is clear that interests and incentives of the research enterprise and the "change agents" within the health care system must be more effectively aligned in order to achieve the ambitious promise envisioned when AHCPR was established.