Evaluation of AHRQ's Children's Health Activities
"What we really want to get at is not how many reports have been done, but how many people's lives are being bettered by what has been accomplished. In other words, is it being used, is it being followed, is it actually being given to patients?... [W]hat effect is it having on people?"
Congressman John Porter, 1998, Chairman, House Appropriations Subcommittee on Labor, HHS, and Education
Since its inception, the U.S. Agency for Healthcare Research and Quality (AHRQ, or the Agency) has funded research and other activities concerning a variety of children's health issues. At the same time, the Agency's strategic priorities continue to evolve in the context of a changing policy environment. Currently, the Agency is crafting strategic plans for its ten "portfolios of work." As part of the planning process, the Agency is evaluating the various cross-cutting, or cross-portfolio, priority populations, including children. To aid in this assessment, the RAND Corporation was contracted to evaluate AHRQ's child health activities.
As requested by AHRQ, this evaluation addresses four Primary Objectives:
- Measure and assess to what extent the Agency contributed and disseminated and/or translated new knowledge.
- Measure and assess to what extent AHRQ's children's healthcare activities improved clinical practice and health care outcomes and influenced heath care policies.
- Measure and assess AHRQ's financial and staff support for children's health activities.
- Measure and assess to what extent the Agency succeeded in involving children's health care stakeholders and/or creating partnerships to fund and disseminate key child health activities.
To address these objectives, RAND developed and implemented a conceptual framework based on state-of-the-art research evaluation theories (Roessner, 2002). This framework suggests that the effect of any particular activity is likely to be highly indirect and that policy and practice outcomes are multi-determined and depend on a loose network of actors (funders, researchers, policy makers, the public, regulatory officials, practitioners). It also suggests that the impact of activities is typically characterized by a highly skewed distribution, with a relatively small number of activities having a large impact and the vast majority having a smaller impact. This framework allowed an assessment of the Agency's contribution to the development and dissemination of new knowledge and the impact of that knowledge on policies, clinical practice, and health outcomes. In addition, we used databases developed at RAND to describe the Agency's funding focus and financial investment in children's health over time.
Pursuant to Primary Objective 1, we analyzed funding for external activities and publications related to those activities, overall and by category, and over time. In terms of funding, we identified over $350 million that AHRQ used to fund external research and activities related to children between 1990 and 2005. The overall amount and average awards rose during the time period, despite a decline in the last few years, as did the percent of the AHRQ overall budget devoted to children's health activities. Examining external activities by AHRQ strategic goals, children's health strategic goals, and AHRQ portfolios of work shows that, over time, activities related to patient safety and health information technology have increased from almost zero to about half of all activities, reflecting trends in the Agency as a whole.
We also tracked publications derived from these activities over time. In terms of AHRQ strategic goals, 54% of the publications describe efforts to improve health outcomes and 31% relate to the goal of improving access, appropriate use and efficiency. In terms of children's health strategic goals, 64% of the publications fit into the goal of contributing to new knowledge about child health services. In terms of AHRQ portfolios of work, 41% of the publications are categorized under the care management portfolio and 31% describe results from projects on the cost, organization, and socio-economics of health care. And because publications lag activities by several years, in the near future, we expect an increasing number of publications regarding health information technology and patient safety. We note that gathering the data required for this analysis was difficult and time-consuming—there is no central repository of information about all funding and all products that can be used for ongoing monitoring of priority populations.
For Objective 1, therefore, our evaluation shows that the Agency has contributed a substantial body of new knowledge as a result of its funding for children's health research (extramural and intramural) and has disseminated this new knowledge effectively in the peer reviewed literature. This analysis also shows that the child health portfolio has changed over time, reflecting the overall Agency priorities.
It is difficult in any field to trace a direct line between research or other activities and improved health care activities or clinical practice, or between such activities and an influence on health policy. Nevertheless, pursuant to Primary Objective 2, we found that the 794 external and internal publications arising from AHRQ's children's health activities were cited nearly 3,000 times in the scientific literature. Similar to previous analyses by AHRQ staff on the results of outcomes research (Stryer, Tunis et al. 2000), we found that 70% of the publications present research findings that may inform policy and practice, but do not directly address particular decisions. Fourteen percent of the publications focus on the impact of policy changes or have clear policy implications. Fifteen percent of the publications describe research that evaluates clinical behavior, demonstrates changes in clinical behavior, or demonstrates the use of tools in a clinical setting. One percent of the publications describe research on the relationship between clinical or policy changes and health outcomes.
Our case examples and key stakeholder interviews, which focused on AHRQ's work in State Children's Health Insurance Program/Child Health Insurance Initiative (SCHIP/CHIRI™), attention deficit hyperactivity disorder (ADHD) and asthma illustrate several lessons on maximizing the impact of AHRQ activities. First, the impact of AHRQ's activities (dissemination of relevant information to policy-makers in a timely manner, changes in practice and outcomes) is maximized when structures are in place to encourage cooperation and communication among researchers and with a variety of stakeholders. The SCHIP/CHIRI™ case illustrates this point: the request for proposals (RFP) was structured to require researchers to work with policy makers; AHRQ partnered with other funders; and the individual projects were required to set aside funds to support initiative-wide meetings and publications. In the case of ADHD and asthma, one of the main impacts of AHRQ involvement was the ability to draw on the resources of Centers for Education and Research on Therapeutics (CERTs) to synthesize evidence, of practice-based research networks (PBRNs) to change practice, and of tools to improve outcomes.
Second, in many of the child health activities funded by AHRQ, impact all too often relied on the individual efforts of Principal Investigators and AHRQ staff rather than being programmed into the activities. With some notable exceptions (for example, CHIRI™), activities are seen as focused on generating products for academic journals. And, the AHRQ infrastructure was seen as being oriented towards the 'front end' of research grants—soliciting, reviewing, selecting, and funding—rather than the 'back end'—disseminating timely and relevant information to policy makers, documenting impact on clinical practices or outcomes, tracking and compiling the ways that AHRQ products are used by various stakeholders.
Third, the inherent tension in serving the needs of multiple stakeholders can be positive or negative. In the case of CHIRI™, tension lead to a creative solution that enhanced the impact of this set of activities. In other cases, the tension was not addressed as successfully. Many interviewees alluded to the tension between the role of the Agency in funding policy-relevant research versus directly informing the policy debate. Child health activities at AHRQ were seen as being more focused on the former, rather than the latter. Another tension is between the needs of academicians to generate peer-reviewed publications on the one hand, and the needs of policy-makers, clinicians, and families for timely, actionable information on the other. Clinicians and families saw AHRQ's child health activities as being more focused on the needs of academics rather than on those of the users. A further tension is between the broad spectrum of needs in children's health and a very limited budget. Specific, focused investments designed to generate spectacular answers to the "Porter Question" (What effect is the research having on people?) were seen as key to the Agency's continued viability.
In sum, for Objective 2, our bibliometric analysis, case studies, and key stakeholder interviews suggest that children's health care activities at AHRQ, along with other child health stakeholders, have played an important role in improving clinical practice and health care outcomes and in influencing specific heath care policies. They also suggest ways to enhance the impact of future children's health activities.
AHRQ's objectives and budget have changed over the years, both overall and for children's health activities. A tightening budget and a shift away from investigator-initiated external research grants have highlighted questions regarding the place of children's health within AHRQ's overall mission. Pursuant to Primary Objective 3, two recurrent themes regarding AHRQ support for children's health activities emerged from the key informant interviews. The first was the issue of whether children's health should be considered (and funded) separately from other activities. Several interviewees made a cogent case for theoretical, practical, and ethical reasons to consider children separately, and made specific suggestions for how to accomplish this. The countervailing sentiment was that, in an Agency with a relatively small budget, setting aside specific funding for children's health activities was likely to result in a children's health budget insufficient to accomplish any of the Agency's children's health strategic goals. Nevertheless, there are specific issues that are inherently child-only, such as research on SCHIP, for which it is appropriate to argue for set aside funding.
The second theme was the widespread perception that children's health activities at AHRQ were dependent almost entirely on the efforts of individual Agency staff. Interviewees noted that there were virtually no formal structures or policies at the Agency to ensure that children's health was adequately addressed; the Child Health Advisory Group (CHAG) does not serve this function. Interviewees praised Denise Dougherty, as Senior Advisor on Child Health, and Lisa Simpson, the former Deputy Director, for their unflagging efforts to raise the profile of children's health, and noted that compared to other special populations, children's health has a more organized and effective constituency within the Agency. However, they also noted that the Office of Senior Advisor has neither authority nor funds to ensure inclusion of these special populations and relied almost entirely on personal persuasion. They also noted that, with Lisa Simpson's departure, high-level advocacy for children's health had diminished markedly. Moreover, within Agency Offices and Centers, it was up to individual staff to ensure that children's health was included in activities. Interviewees lamented that there did not seem to be an institutional-level, coherent voice for child health at the Agency and wondered aloud what would happen to children's health research at the national level if AHRQ did not continue, in some important way, to lead.
Therefore, for Objective 3, our interviews and case studies show that although individual Agency staff have performed heroically in assuring continued support for child health activities, there is a lack of authority or resources devoted to children's health that has limited AHRQ's financial and staff support for children's health research.
Pursuant to Primary Objective 4, interviewees perceived AHRQ as having been successful in nurturing a growing children's health services research community, through National Research Service Award (NRSA) fellowships, conferences, and training grants. But they also noted that this same community is in danger of dissolving or moving to other topics of inquiry as funding in this area continues to be scarce. In terms of partnering with other HHS entities, key stakeholders perceived limited success. Interviewees at other HHS entities professed little experience or interest in partnering with AHRQ, despite substantial respect for the AHRQ personnel that they had interacted with. As a result, AHRQ's partnerships with other HHS entities tend to involve specific initiatives or committees and are initiated or maintained at the program or project-officer level. Interviewees could not cite examples of high-level inter-agency collaboration in children's health. In terms of partnering with policy entities, CHIRI™ was a good example of successful partnering, but interviewees noted that there is little interest in children's health policy at the federal level. Several interviewees suggested that AHRQ ought to further collaborate with state-level policy makers and with the Centers for Medicare and Medicaid Services (CMS) around SCHIP and Medicaid. Interviewees reported mixed success in partnering with the clinical and patient/family communities. They suggested that far more could be done to partner with providers through professional societies such as the Association of American Pediatricians (AAP) and other organizations involved in improving children's health care.
Thus, for Objective 4, we found that AHRQ staff has pursued numerous connections with other agencies, but primarily through participation on committees and task forces, both within and beyond HHS. The Agency has had mixed success in involving children's health care stakeholders and/or creating partnerships to fund and disseminate key child health activities.
Building on this analysis, we offer the following suggestions for the Agency's consideration:
Through their children's health activities, AHRQ has successfully created and disseminated knowledge and engaged the pediatric academic community, but has been less successful in creating and documenting improvement and in engaging the children's health quality, practitioner, and patient/family community. Answering the Porter Question for children will require AHRQ's child health activities to focus more effectively on identifying and pursuing opportunities to apply existing evidence, implement change, and document improved processes and outcomes. It will also require the Agency to work more closely with clinical organizations such as the American Academy of Pediatrics (AAP) and with organizations organized around quality improvement in order to better leverage the strengths of each. Experience with the CHIRI™ program as well as with AHRQ's work on asthma and ADHD show that this is possible, at least in well focused areas. The CHIRI™ program required (and facilitated) collaboration between researchers and policymakers and in that way could be a model for collaboration. However, it was organized around an ongoing federal project requiring reauthorization. In the absence of a similar national quality improvement project, it would be necessary to pursue a more piecemeal approach. We note that, with the restriction of funding for investigator-initiated grants and increasing prescriptiveness from HHS regarding what funds can be used for, fewer opportunities exist for pediatric quality improvement research.
The fate of children's health at AHRQ seems to rest in the hands of a few extremely dedicated individuals who are forced to rely on their powers of personal persuasion. In order to institutionalize child health as an Agency priority, certain structural and procedural changes are required. A more formal and rigorous monitoring of Agency funding commitment by priority population should be instituted, with regular presentations to Agency leadership about children's health activities and identification of under-funded areas. Endowing the Senior Advisor for Children's Health with institutional authority (for example, participating in AHRQ staff employee performance reviews) or with small amounts of discretionary funds to pursue promising opportunities to answer the Porter Question would substantially enhance effectiveness. AHRQ should also examine how to increase the effective use of the contracts mechanism to pursue focused efforts to answer the Porter Question.
Given AHRQ's limited budget, partnering/leveraging with other HHS entities is critical. The initiative for partnering must come from the highest levels within the Agency, rather than relying on program-officer level contacts. AHRQ should aggressively pursue opportunities to collaborate with CMS around SCHIP and Medicaid. We note that AHRQ staff are currently engaged in collaborative efforts with CMS around child health issues and that the Agency's authorizing language must guide collaborations.
Finally, we note that all of these analyses required a substantial effort on our part to identify the AHRQ research and projects that were related to child health, as well as to identify their outputs and effects. AHRQ should consider building a management information system (or modifying existing systems) to reliably track the inputs, outputs, and impacts of its child health activities so that this does not have to be done on an ad hoc basis as for this study. Such an infrastructure would provide a coherent and consistent picture of children's health activities at AHRQ, which would be a useful tool for both management and evaluation purposes. The information system ought to be focused on tracking dissemination and impact as well as getting through the grant review and funding process.