Chapter 3. Results
Evaluation of AHRQ's Children's Health Activities
Primary Objectives 1 and 3
Primary Objective 1 was to measure and assess to what extent the Agency contributed new knowledge as a result of its funding for children's health research (extramural and intramural) and disseminated and/or translated effectively its findings to meet AHRQ's strategic objectives of improving the safety, quality, effectiveness and efficiency of health care as well as wider DHHS strategic objectives. Primary Objective 3 was to measure and assess AHRQ's financial and staff support for children's health research as well as Agency internal handling of children's health grants, contracts and intramural activities research with/among other AHRQ programs, portfolios and activities and other DHHS and federal agency efforts.
To address these objectives, we examined the number of activities, level of funding, and articles produced. We first describe these over time, then by AHRQ Strategic Goal, by AHRQ Children's Health Strategic Goal, and by AHRQ Portfolios of Work. We also report the views of key informants with respect to Objective 3. All of the tables cited in this section appear in Appendix C.
Children's health activities over time
Using the methods described in the preceding section, we identified a total of 514 external activities related to children's health. These included grants or contracts for specific research projects in addition to grants for conferences, meetings, workshops and centers. Looking at the number of activities across a 16 year time period shows the variation across time that reflects changes in priorities, staff and funding (Table 3). We then looked at the distribution of external activities across different time periods: 1990-1995, 1996-1998, 1999-2002 and 2003-2005 (Table 4). We somewhat arbitrarily divided the time periods based on events of importance for children's health at the Agency. For example, 1996 was the year that Lisa Simpson was appointed Deputy Director, 1999 was the year the Agency was reauthorized as AHRQ, and 2002 was the year Lisa Simpson left the Agency. AHRQ funded an average of about 14 external activities from 1990 through 1995. The average rose slightly to 18 external activities per year in the period of 1996-1998. Starting in 1999 (the year in which the Agency for Health Care Research and Policy was reauthorized as AHRQ) and lasting through 2002 the average rose substantially to 64. AHRQ funded an average of just over 39 external activities per year from 2003 to 2005.
Figure 1 shows the total number of external activities by year with the average for the time period noted by the dashed line. Even within the time periods, there was considerable variation from year to year in the number of external activities funded in a particular year. Nonetheless, and notwithstanding the relative decrease in the last time period, the number of external activities rose considerably over the time period studied.
As described in the preceding section, we obtained funding information for 426 of the 514 external activities. We were unable to find funding information for most of the activities initially funded in 2005, for those related to the evidence based practice centers, for contract work and for a few others of the external research activities. Excluding these, we identified more than $350 million for external research and activities related to children between 1990 and 2005. The total funding for this subset of the external activities across the 1990 to 2005 time period ranges from less than $5 million per year to more than $55 million per year (Figure 2). Looking at the total funding across the time periods of interest, the majority (59%) of the funding for occurred between 1999 and 2002 (Table 5).
Given the variation in the number of activities per year, we also examined the average funding per external activity over the life of the grant by the initial funding year (Figure 3). Overall, the funding varied from $10,000 for faculty development awards or capacity building grants to more than $5 million for different quality, patient safety and HIT activities like CAHPS® or CERTS. From 1990 to 1993 the average award rose from close to $400,000 to over $600,000 before dropping back to around the 1990 level by 1995. In 1996, the average funding per external activity rose markedly to over $900,000 before a sharp decline in 1997 to just over $200,000. The average award approached $1 million in 1998 before going over in 1999. By 2005, the average award had stabilized somewhat at just under $1 million. The variation in the average award reflects the start of different initiatives such as CHIRI™.
We also examined the external activities related to children as a percentage of the total AHRQ budget (Figure 4). Using information from AHRQ budget justification documents found on the AHRQ Web site, we found the total funding for AHRQ during the time period of interest.17 Figure 4 shows the percentage of the total AHRQ budget spent on external activities related to children. Between 1990 and 1995 external activities related to children averaged just 6 percent of the AHRQ budget. The percentage rose somewhat to an average of 9 percent between 1996 and 1998. Again, the 1999 to 2002 time period shows dramatic changes with the average percentage of external activities related to children rising all the way to 23 percent of the AHRQ budget. From 2003 to 2005 the average fell again to 7 percent of the AHRQ budget. Note that the child health activities are categorized by first year of funding, which the overall AHRQ budget in any year includes projects that started at some time in the past, so the comparison is only approximate.
Overall, the funding analyses shows that the amount of funding and average awards for external activities related to children rose during the time period, but that there was no actual increase in the percent of the total AHRQ budget dedicated to these activities because the AHRQ budget increased at a similar pace.
Recognizing that the research projects take several years to complete and that publications often lag behind, we focused our publications analysis on external research projects that were funded from 1996 through 2002. Our bibliometric search (described in section 2) identified a total of 749 publications from the 258 AHRQ children's health activities funded between 1996 and 2002.18 We first looked at the distribution of these publications by the initial funding year of the grants (Table 6). Since the number of activities varied from year to year, we also calculated the average number of publications per activity for each year. The activities funded in 1996 averaged 3.6 publications per project. The average fell in 1997 and 1998 before rising substantially in 1999 to an average of 4.4 publications per project. The pace of publications fell again in 2000 to an average of 3.4 per activity. While projects funded in 2001 and 2002 averaged less than two publications per project, the lower numbers likely reflect the time needed for research and publication rather than the productivity of the activities per se. Over the time period examined, the external activities related to children's health averaged almost three publications per activity.
We categorized the external activities using three different typologies: AHRQ strategic goals, AHRQ children's health strategic plan goals, and AHRQ portfolios of work. For the AHRQ strategic goals we looked at the Strategic Plan released in 199819 and the FY 2006 Budget Justification.20 Both documents lay out strategic goals for the agency (Table 7). AHRQ developed the three Strategic Plan goals to carry out its mission. In developing the Budget Justification goals, AHRQ looked across its entire research portfolio and created categories to describe different groups of activities. The first three of these goals have been in place since the FY 2000 budget justification. The fourth goal was added in the FY 2005 budget justification. We have not included the fourth goal in our categorization scheme since it relates to AHRQ's internal processes.
We combined these two sets of strategic goals into a single set of five AHRQ strategic goals. Table 7 shows how the two sets of goals map into the consolidated set of AHRQ strategic goals. Since both sets had a goal related to access we grouped those together. Otherwise we included the other goals from each set separately. We then looked at how the external activities distribute across these strategic goal categories (Table 8). Overall, over one-half of the activities (52%) fall into the area of supporting improvements in health outcomes. This category includes a wide variety of grants on the causes and effects of specific medical conditions, risk factors and characteristics of those with specific medical conditions, and patient behaviors. This category also includes research on the effectiveness of interventions and establishing an evidence base in a particular area. Research activities that examined disparities in outcomes related to race/ethnicity, income level, location or risk status were also included in this category. We classified activities related to developing curricula, tools, decision aids or guidelines in this strategic plan area. Training related activities such as fellowships, faculty research programs and research collaborations also fell into this category.
One-quarter of the external activities related to health care utilization and access to care. These included research on utilization and cost effectiveness of SCHIP, Medicaid, and HMOs. Ten percent of the external activities related to technology and the development of information systems, including studies on medical informatics, electronic interfaces, telemedicine, electronic medical records and other health information technology projects. Another 11 percent of the projects focused on patient safety, including research on medical errors and quality of care. Just two percent of the activities focused on strengthening quality measurement and quality improvement.
Over time, the distribution of activities across these strategic goals shifted in some cases by substantial amounts. The proportion of activities related to improving health outcomes declined from 63 percent during 1990-1995 to 38 percent for the 2003-2005 time period. Over that same period, the percentage of external activities focused on improving access, appropriate use and efficiency declined from 37 percent to 13 percent. The relative emphasis on health information technology and data driven projects increased from zero to 31 percent of external activities. A similar pattern emerges when looking at the safety goal, where the percentage rose from 8 percent to 18 percent over time.
We also looked at the how the funding was distributed across the AHRQ Strategic Goals (Table 9). Among the subset of external activities with available information on funding, 40 percent of the total funding went to projects aimed at improving health outcomes. Nearly one-quarter (24%) of the total funded projects went to promote patient safety. Another 21 percent of the total funding went toward external activities to improve access, appropriate use, efficiency and costs. Overall, the funding for activities on data and health information technology represented 13 percent of the total, including studies on database development, medical informatics, telemedicine, electronic medical records and other health information technology projects. Relatively little (1%) of the total funding went toward activities for quality measurement and improvement.
When comparing the distributions of funding and activities across the AHRQ Strategic Goals, we see that relatively more funding went to promoting safety (24% of funding v. 11% of activities) and less to improving health outcomes (40% of funding v. 52% of activities). These differences presumably reflect both the higher average costs of safety projects, and the increase in the average cost of projects over time (Figure 2), since these categories were more prominent later in the period.
We categorized each publication using the different categorization schemes by assigning the publication to the category of the external activity from which it derived (Table 10). More than one-half of the publications (54%) describe efforts to improve health outcomes. Nearly one-third of the publications (31%) relate to the goal of improving access, appropriate use and efficiency. Ten percent of the publications from external activities involve promoting patient safety. Relatively few of the publications fall into the strategic goal categories on using data to make informed decisions (3%) and strengthening quality measurement and improvement (2%). Overall, the distribution of publications across the AHRQ strategic goal areas tracks closely the distribution of the activities across these areas. There were somewhat more publications than activities on improving access to care and efficiency (31% v. 25%) and fewer publications on health information technology and data use (3% v. 10%).
Looking at the analyses by AHRQ strategic goals more broadly, we see that the majority of activities fit into the categories intended to "improve health outcomes" and "improve access, appropriate use and efficiency and reduce costs." Over time, however, the proportion of activities addressing these two strategic goals decreased from 100 percent of the children's health activities to 51 percent. The number of activities addressing the strategic goals related to safety and health information technology together make up 22 percent of the entire portfolio, but have increased from 0 to 49 percent over time. The overall funding picture for the AHRQ strategic goal areas largely reflects this trend with a relatively larger proportion of the funding going towards activities in the safety area. As noted above, the publications more closely tracked the distribution of activities with the majority of publications coming from activities in the areas of improving health outcomes and improving access, appropriate use and efficiency and reducing costs.
AHRQ children's health strategic goals
For the second categorization scheme, we looked at the 1999 strategic plan document that designated children as a priority population.21 In that document, AHRQ identified a set of six goals for their children's health agenda:
- Contribute to new knowledge about child health services.
- Create tools and nourish talent to strengthen the knowledge base in child health services.
- Translate new knowledge into practice.
- Improve communication with stakeholders in child health.
- Include children and child health care in all AHRQ-supported research, as scientifically and ethically appropriate.
- Balance the AHRQ research portfolio to represent a broader range of children and child health care.
Looking at the distribution of external activities related to children's health across these six goals, nearly two-thirds (61%) of the external activities related to contributing new knowledge about child health services, including activities on outcomes, quality, safety, cost, utilization and access to care (Table 11). One-quarter of the external activities focused on creating tools and supporting researchers involved in children's health services research. This category includes activities designed to develop instruments, tools or guidelines as well as those supporting research networks and programs. The health information technology, database development, electronic medical record, and telemedicine activities also fall into this category.
The other four children's health agenda goals were considerably less common. Eight percent of the external activities were aimed at representing a broader range of children and child health care. Here, we included activities that address disproportionality, disparities and vulnerable populations. Six percent of the external activities related to translating new knowledge into practice. Many of these were evidence reviews carried out by the AHRQ-funded Evidence-Based Practice Centers. While many of the external activities could be categorized as including children and child health care in all AHRQ-supported research, we used this category for only a handful of activities, preferring to categorize the activities according to the subject matter. The few activities that fell into this category related to public health issues such as hospital disaster plans and surge capacity that effect children as well as adults. None of the external activities were categorized as primarily related to improving communication with stakeholders in child health. While AHRQ funded a number of conferences and workshops to bring stakeholders together, these activities were categorized according to their topical area.
Over time, there were notable shifts in how the external activities fit into the different children's health agenda goal categories. From 1990 to 1995, nearly three quarters (72%) of the external activities related to contributing new knowledge about child health services. By the 2003-2005 time period, the percentage had fallen to 45 percent. Over the same time period, more of the activities fell into the category for creating tools and nourishing talent to strengthen the knowledge base with the percentage rising from 15 percent in 1990-1995 to 44 percent in 2003-2005. The percentages for the other children's health agenda goals remained fairly consistent over time.
We also looked at the distribution of total funding across the Children's Health Strategic Plan Goals (Table 12). Among those activities with funding information available, 55 percent of the total funding went to activities designed to contribute to new knowledge about child health services. More than one-quarter (28%) of the total funded external activities were aimed at creating tools and nourishing talent. Nearly equal percentages of the total funding went to projects to translate evidence into practice (8%) and to include a broader range of children in AHRQ research (9%). Very little of the total funding (1%) went to projects that fit into the category on including children and child health care in all AHRQ-supported research. As we mentioned earlier, while many activities fit into this overarching goal area we categorized the activities by subject matter so few activities fell into this category. Overall, the distribution of funding across the Children's Health Strategic Plan Goals closely tracks the distribution of activities.
We also categorized the publications coming out of AHRQ's child-related external activities according to the Children's Health Strategic Plan Goals (Table 13). Almost two-thirds (64%) of the publications fit into the category on contributing to new knowledge about child health services. Nearly one-fifth of the publications (18%) related to the goal of creating tools and training investigators. Nine percent of the publications described translating knowledge into practice. Another nine percent related to broadening the range of children included in research. The distribution of publications across the Children's Health Strategic Plan Goals closely mirrors the distribution of the external activities described earlier.
An overall picture of the Children's Health Strategic Plan Goals shows that most of the activities sought to "contribute to new knowledge about child health services" and to "create tools and nourish talent to strengthen the knowledge base in child health services". Over time, however, the breakdown of activities in each of these categories changed dramatically from a relative imbalance (72% v. 15%) to nearly equal (45% v. 44%). The distribution of funding and publications across the Children's Health Strategic Plan Goals closely tracks the overall distribution of activities with 61% related to contributing new knowledge and 25% related to creating tools and nourishing talent.
Overall, 37% of the external activities related to children's health funded by AHRQ from 1990 to 2005 fall into the care management portfolio (Table 15). These included research projects on specific medical conditions, health disparities, outcomes, quality improvement and the development of instruments, tools, and guidelines to aid clinical practice. Twenty-six percent of the children's health research is related to the cost, organization, and socio-economic portfolio of research. This research focuses on utilization, access and cost effectiveness and includes the work on SCHIP and Medicaid.
Ten percent of external activities related to the health information technology mission, including those on medical informatics, electronic medical records, and bar code technology. A few additional activities specifically related to database development were categorized into the data development portfolio of research. Eleven percent of the activities fall into the patient safety portfolio. Overall, just six percent of the activities addressed the prevention portfolio which focuses on evaluating effectiveness and promoting evidence-based practice. Another seven percent of the activities are categorized in the training portfolio. These include some fellowship and conference grants as well as grants to support faculty research programs. Only a few of the children's health related activities related to the pharmaceutical outcomes portfolio (2%) and the system capacity and emergence preparedness portfolio (1%).
Looking at the distribution of activities over time, there are some notable shifts in the relative emphasis on the different portfolios of research. Two of the portfolios decreased notably. By the 2003-2005 time period, activities categorized in the care management portfolio had decreased from 50 percent to 26 percent of the total. Similarly, research projects in the cost, organization, and socio-economics portfolio declined from 38 percent to 13 percent. Two other portfolios gained substantially. From 1990 to 1995 there were no health information technology projects related to children's health at AHRQ. By the 2003-2005 period, one-third of the external activities involved health information technology research. Similarly, the patent safety portfolio grew from zero to 16 percent of all external activities. The percentages for the other portfolios of work remained fairly consistent over time.
As with the other categorization schemes, we also examined the distribution of total funding across the AHRQ Portfolios of Research for those activities with available funding information (Table 16). Nearly equal percentage of the total funding went toward external activities in the care management mission (23%) and cost, organization, socio-economics mission (22%). Eighteen percent of the total funded external activities on patient safety, while 14 percent funded health information technology work. Less than ten percent of the total funding was directed toward external activities in the training mission (9%), prevention mission (7%), pharmaceutical outcomes mission (6%), data development mission (<1%) and system capacity and emergency preparedness mission (<1%). When comparing the distribution of funding to the distribution of activities, a few of the portfolios of research represent more of the total funding than the activities. For example, external activities related to health information technology represent 14 percent of the funding and 10 percent of the activities. Likewise, patient safety activities are 18 percent of the total funding and 11 percent of the activities.
The distribution of publications according to the AHRQ Portfolios of Research shows that a substantial minority (41%) of the publications were categorized under the care management mission which includes research on specific medical conditions and health outcomes (Table 17). Nearly one-third of the publications (31%) describe results from projects on the cost, organization, and socio-economics of health care. Nine percent of the publications relate to patient safety and another 9 percent relate to prevention. Relatively few of the publications fell into the portfolios for health information technology (3%), pharmaceutical outcomes (4%) and training (3%). The distribution of publications across the Portfolios of Research looks quite similar to the distribution of the external activities described earlier.
Looking across all of the analyses related to the Portfolios of Research, we see the same kind of trend toward safety and health information technology that we saw with the AHRQ strategic plan goals. Overall during the study period, 37 percent of activities were related to the care management mission and 26 percent were related to the cost, organization, and socio-economics mission. Over time, however, the proportion of activities addressing these two strategic goals decreased by nearly half from 88 percent of the children's health activities to 49 percent. The number of activities addressing the health information technology and patient safety missions each make up about 10 percent of the entire portfolio, but have increased from 0 to 39 percent over time. For the funding, we see that external activities related to patient safety and health information technology represent more of the total funding than of the total activities. As noted above, this pattern is not seen with the publications where the distribution of publications mirrors that of the activities.
Key informant interviews regarding AHRQ support for children's health activities
Many interviewees shared a similar view of AHRQ's support for children's health activities. They were grateful for the support that had been given and praised AHRQ staff for their tenacity in championing child health issues. However, they were frustrated that, rather than being built into the structure of the organization, children's health activities were supported exclusively to the degree that AHRQ staff devoted time and effort to them. As one interviewee reported,
" AHRQ deserves a lot of credit for what they have done and Lisa (Simpson) and Denise (Dougherty)'s work has been great. But they need to institutionalize it rather than being dependent on one person. Children's health needs to become part of a checklist that everyone goes through. Whatever activity or project someone is working on, they should be looking from the beginning at whether there is a children's component."
This was the case for external research activities as well as for intramural research and internal functions. Thus, RFP development, study section composition, decisions at the funding committee level, reports generated by intramural researchers, participation in the Children's Health Advisory Group (CHAG), and initiatives at various Centers all depended on the effort of individual AHRQ staff members. Respondents praised Denise Dougherty and Lisa Simpson23 for their efforts to bring children's health to the fore and to make sure that children were represented in all AHRQ activities. Several noted that, compared to other priority populations, children's health was successful in maintaining its profile at AHRQ. However, respondents noted that without specific authority or resources devoted to children's health or vested in the position of Senior Advisor on Children's Health, virtually all children's health activities relied on the personal influence of a small number of AHRQ staff members.
Many respondents acknowledged the difficulties in integrating children's health into the larger agenda and of raising its stature in the debate over health care. As one interviewee noted,
" There is not a coherent voice about children's health and health care issues at the policy level at AHRQ. There is no clear receptor site for those activities. Even with the listserv and other kids' activities, there still is not a coherent view. No one has figured out how to talk about the children's quality agenda and AHRQ has not devoted a lot of resources to area. It's starting to look at these issues but only with outside prodding. "
Similarly, respondents noted that AHRQ's budget for external research was extremely limited and that much of it was earmarked for specific initiatives that were more relevant to the Medicare population than to children. Reflecting this, several respondents worried that children's health services research may be critically endangered, since "if they (AHRQ) don't do it, then who will?" This is especially true given that no other federal agency has the same kind of mandate as AHRQ.
Part of the integration difficulty is reflected in the fact that creating a cohesive children's health activities portfolio is challenging. As another respondent reported:
" There is a strong children's health portfolio but not necessarily something that naturally comes together (like health IT) to show what you have learned. With children, AHRQ seems to focus on funding children's health research generally without identifying what areas are most important. As a result, at the end of the day, they look back through the research that has been done and try to create a cohesive whole instead of creating a focus from the outset which would then result in a cohesive story"
17. Appropriation History Table. February 1998. Rockville, MD: Agency for Health Care Policy and Research. https://www.ahrq.gov/about/cj1999/apphis99.htm; Justification for Budget Estimates for Appropriations Committees, Fiscal Year 2005. February 2004. Agency for Healthcare Research and Quality, Rockville, MD. https://www.ahrq.gov/about/cj2005/cjweb05.htm.
18. The total number of activities in the publications analysis excludes those external activities listed as conferences, meetings, workshops, trainings or centers since these were not designed to produce publications in the peer-reviewed literature.
23. We note that, while Lisa Simpson's support was certainly instrumental, it was not the only factor in bringing child health activities to the fore: the efforts of CHAG, as well as other Agency staff, and the emphasis on some aspects of child health from the Secretary's level and the White House (e.g., SCHIP, childhood asthma), and the presence of funding for investigator-initiated grants all contributed as well.