Skip Navigation Archive: U.S. Department of Health and Human Services www.hhs.gov
Archive: Agency for Healthcare Research Quality www.ahrq.gov
Archive print banner
Evaluation of AHRQ's Partnerships for Quality Program

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: https://info.ahrq.gov. Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to www.ahrq.gov for current information.

Chapter IV. What Did the PFQ Projects Achieve?

AHRQ sought projects that aimed to make a "significant improvement in quality of care for a substantial part of the population of the United States. AHRQ is seeking projects that will, in aggregate, affect the quality of care of patients numbering in the hundreds of millions."  (PFQ RFA, May 2002)  This chapter assesses the achievements of the PFQ grantees over the course of their projects.  After a brief overview of the project's overall outcomes, it reviews the experiences and results of all 20 grants by areas of common focus.  

A. Overall Outcomes

For a program with limited visibility, PFQ does appear to have made a difference in health care security, quality and safety in some of the targeted health care organizations, and raised quality of care processes and outcomes for many Americans. Though final outcomes are not known for all projects, it appears that some projects achieved better results than others (Table IV.1). 

In terms of their ability to change clinical practice in ways consistent with evidence, four projects stand out based on the magnitude and scope of their effects: 1) Child Health Corporation of America, which improved clinical performance in several areas at 18 hospitals and has expanded quality improvement efforts at 42 children's hospitals; 2) International Severity Information Systems, which streamlined care processes in nursing facilities in ways that led to demonstrated reduction in pressure ulcers; and has launched a follow-up project to spread its approach more widely; 3) Physician Micro Systems/MUSC, which has expanded an effective strategy to get performance data into greater use in physician offices for improved process of care; and 4) the Visiting Nursing Service of New York, whose model for diabetes home care has shown positive effects and is being extended in 10 states.  

Though less striking, four other projects developed new approaches to quality improvement that have the potential for attaining broader scope and merit greater attention:  1) the American Academy of Pediatrics, which has sustained its clinical improvement efforts through new projects that build on its practice-based, quality-improvement CME course, and has linked the approach to board certification; 2) the American College of Physicians, which had strong preliminary results in diabetes care improvement and is pursuing team-oriented CME projects in other clinical areas; 3) the AMA, which is now working with EMR vendors to integrate its performance measures into their systems; and 4) Catholic Healthcare Partners, whose work on improving heart failure care in hospitals is promising and is being disseminated nationally through the American Heart Association.  

Other grants effectively pursued important areas but did not generate detectible positive improvements, though they have important lessons to share within their respective fields. For example, The Leapfrog Group's work on performance incentives may well be very important in enhancing understanding of the barriers to introducing these incentives. The Lehigh Valley Hospital and Health Network's approach to diabetes control proved it was financially feasible for primary care physicians, but little was done to replicate it beyond the 10 small practices where it was tested. Similarly, the Association of California Nurse Leaders work on falls prevention, though ultimately disappointing in its results, was important and will likely enhance support for performance monitoring in other clinical areas.  Others, like the work by JCAHO, while directed more at building knowledge than seeking immediate changes in practice, may have promise down the road in influencing care.

In the area of bioterrorism preparedness, the tools developed for training physicians in Connecticut were important, even though project leaders found that training had only a short-term effect on physician knowledge.  Findings from the other three bioterrorism preparedness projects may help some local health providers strengthen their plans, and produce new knowledge or tools for health system response planning, but their significance and overall contribution to the field are difficult to assess. 

A few grants, however, did not appear to be well-conceived from the start, even though they were well-intended. For example, the fact that nursing needs to be a focus in improving quality in nursing homes should not have been a surprise to the American Medical Directors Association Foundation. More thought could have been given to the goals and approach behind HealthFront's project, which achieved far less than it originally planned. The impact of RTI's study of the science of partnerships remains difficult to evaluate. 

Return to Contents

B. Outcomes of Projects Seeking to Change Clinical Practice

The concepts of the RE-AIM evaluation framework—reach, effectiveness, adoption, implementation and maintenance—are particularly relevant to assessing the impact of the 17 PFQ grants seeking to affect clinical quality of care.12  The RE-AIM framework is oriented toward assessing the potential for translating research to practice, and for wider dissemination. While this framework can be used to assess interventions at both the individual and organizational levels, in this evaluation we focus on the PFQ projects' effects at the organizational level, since the PFQ projects were intended to scale up proven health care interventions already demonstrated as effective for individuals. This section assesses 17 PFQ grantees' impacts in the RE-AIM framework domains relevant to these projects—reach, implementation, effectiveness, and maintenance/sustainability.13 

1. Reach

When it announced the original 22 projects to be funded, AHRQ stated that they would "involve more than 88,000 medical providers; 5,800 hospitals, nursing homes, and other health care facilities; and 180 health plans."14  Although these estimates were based on overly optimistic predictions at the start of the program, PFQ did not achieve short-term effects on the delivery system on this scale. 

The number of organizations targeted ranged widely across the PFQ projects, even among those targeting the same type of organizations.  (Go to Appendix Table A.4 for a visual display of the number of organizations, patients, or other targets chosen by each project.)  For instance, in projects targeting hospitals for their interventions, the number initially targeted ranged from just a handful (Catholic Healthcare Partners) to between 10 and 40 (CalNOC and CHCA) to 100 (AHA/HRET, Leapfrog) Among those targeting nursing homes, the number targeted ranged from 8 (ISIS) to 30-50 (NYS-DOH and AMDA).  In projects targeting physician practices, the number ranged from 8 (Lehigh Valley) to 10-35 (AMA ACP) to more than 100 (PMSI and AAP).

Projects meeting or exceeding planned reach/participation. Among 17 projects that specified the number of target health care delivery organizations, physician practices, or other local partners they planned to recruit for an intervention, 14 enlisted at least the number of entities projected in their original proposals. This is not an insignificant accomplishment, since few of the projects paid provider organizations anything for participating other than nominal fees to offset the cost of data collection or travel to project meetings.  The only participation incentives project leaders could offer were the free training or technical assistance to improve care quality, and in some cases, the opportunity to learn from others.    

Some projects had low targets, so they attained them easily. For example, ISIS enrolled 12 nursing homes, VNSNY enrolled 8 home health agencies, and Catholic Healthcare Partners recruited 6 hospitals. Other projects set substantially higher targets, but still met them. For example, the PMSI project, conducted with the Medical University of South Carolina, expanded the number of primary care practices participating in its performance measurement system from 40 to about 100. Recruiting the practices was part of the PMSI's regular operations, and participation was relatively easy for provider practices, once they purchased the electronic medical record system sold by PMSI. ACP met its target of about 35 physician practices for its team-oriented, practice-based CME training programs, which required practices to send 3 staff members out of the office to participate in training, implement workflow redesign in their practices, and submit data regularly.

Some projects had to revise their recruitment or research design strategy to reach their target.  For example, when AMDA realized that the best way to gain nursing facilities' participation was by persuading the Director of Nursing, rather than the Medical Director, it switched its focus.  AMDA also loosened its participation criteria and allowed "rolling" enrollment, rather than all at one time.  Even Catholic Healthcare Partners initially had a hard time recruiting its own hospital CEOs to participate in its program, when they couldn't see "clear hospital revenue and profitability gains."  They overcame the CEOs' resistance by asking the system's ultimate decision makers—the nuns who govern the system—to persuade the CEOs to cooperate. 

One project far exceeded the participation target it had originally projected. In the third year of its four-year project, CHCA significantly expanded the number of hospitals eligible to participate in its QI efforts from the original 14 CHCA participating hospitals to all 42 member hospitals. This expansion occurred in part because non-participating sites realized the value of the quality improvement efforts and early PFQ interventions, which coincided with member hospital CEOs recognizing that QI was not just something for the quality department; rather that "quality was the business they were in."   

Projects falling short of planned reach/participation.  Three PFQ projects did not recruit the targeted number of participating organizations, primarily due to difficulty in overcoming barriers to provider involvement. For example, the American Hospital Association-Health Research and Education Trust (AHA-HRET) sponsored a project that worked with seven hospital-based palliative care units to offer on-site visits and support to other hospital teams wishing to develop or enhance their own palliative care units.  This project found that even when most program costs were subsidized, the difficulty of making the business case to hospital administrators dampened interest.  

NYS-DOH did not recruit all of the adult care facilities it planned to participate in its training program, largely because these organizations are not required to provide staff training and resource problems make it hard for them to spare staff to participate.  Long-term care facilities, especially those that are small, appear to be less willing or able than hospitals to take on any "extra" activities, particularly when the incentives or rewards for doing so are long-term or uncertain. The Connecticut Department of Health/Yale New Haven Health System found it very hard to persuade physicians to take its bioterrorism preparedness course, and as a result did not expand the effort to target other groups of professionals or to hospitals and practitioners in other parts of the state as originally intended.  

2. Implementation of the Intervention Model/Strategy

Implementation in the RE-AIM framework refers to the fidelity to the core elements of an intervention protocol, that is that they are implemented consistently with the design or model. In this evaluation framework, the question of fidelity is framed as whether the intervention was delivered as intended. While most grantees were successful in this regard, a few encountered problems that required they modify original plans and adapt models. 

One of these, the American Medical Association project, had to change its strategy significantly from one that planned to test and compare two models for collecting data from physicians on performance measurement, to a focus on just one of the models.  This change occurred after the groups involved in testing the so-called "community model" for collecting data from payers encountered resistance to sharing data on physician quality measures. The project shifted gears to focus exclusively on the "practice model," in which physicians transfer data electronically to a central data repository. In making this change, AMA expanded beyond its original focus to invite a variety of physician practices—a large specialty group, a university-based outpatient group, and a publicly-sponsored ambulatory care network—to test the model and help it learn how different types of electronic health information systems could be adapted to export data for measuring performance against AMA-developed standards.

Also encountering operational constraints, the New York State Department of Health reduced the number of best practices it expected nursing homes and adult care facilities to implement to make it easier for them to participate and increase their ability to train staff in the best practices.

HealthFront also encountered operational problems that challenged the original project concept. Originally hoping to develop a nationally recognized provider performance measurement system, the grantee decided to focus more intensively on supporting purchaser capacity in two markets (Minneapolis and Colorado) after one of the key partners had to withdraw. Key partners in these markets had competing obligations; they supported the work of the grant but couldn't provide the fast response originally assumed. As a result, this project transitioned into a strategy focused more on generating information on how financial incentives to doctors could be aligned and how providers perceived incentives than its original focus on introducing these incentives over the course of the grant. 

Use of IT to support quality improvement. While nearly all PFQ projects collected data from target organizations to track progress and evaluate outcomes, three projects (AAP, ISIS, PMSI) sought to introduce new information technology into facilities or provider practices as a tool for quality improvement or quality measurement.  Several others (AMA, Lehigh Valley, ACP, CHCA, VNSNY) collected data from providers and used third parties to deliver timely reports to provider organizations to provide frequent feedback on the success of quality-related efforts.

Most solved the difficulties of incorporating the new technology or data collection and reporting tools into daily workflow.  But some ran into problems that slowed their progress or caused them to make significant shifts in strategy.  For example, the American Academy of Pediatric's intervention relied on pediatricians' use of a new online tool for reporting care processes, called eQIPP. When the PFQ project began, this tool was still new and not completely reliable. The American College of Physicians found that the data coordinating center it used was slow to produce results needed by the participating physician practices to assess changes in their patients' clinical indicators. 

Adapting interventions to each participating organization/group. Several projects found it challenging to identify essential elements of their intervention versus those that could be modified to adapt to each organization's culture, IT infrastructure and staffing patterns. For example, RTI's project found that many health care innovations are complex and have multiple elements, but evaluations of their effectiveness do not distinguish between elements that are required or optional.  ACNL/CalNOC's project allowed each hospital to select which evidence-based practices to implement to reduce hospital-based falls, but when its results did not show a significant reduction in hospital falls or falls-with-injury, variation in the interventions may explain the lack of impact.

3. Effects on Health Care Delivery Processes and Clinical Outcomes

Of the 17 grants focused on health care quality or patient safety, 12 set measurable goals related to change in clinical practice or outcomes.  Of these 12, 8 had preliminary results to report by September 2006.15 Go to Appendix Table A.5 for a brief summary of all projects' preliminary outcomes.  All but one of the eight detected some improvement in the measures examined, suggesting the majority were at least somewhat successful.  However, the magnitude of the changes is not consistent across measures and in some cases, is difficult to assess from the information provided by project staff. 

  1. The American College of Physicians examined process of care measures, such as eye and foot exams and flu vaccines, and clinical outcome measures, such as blood pressure, LDL below recommended levels, and so on among patients with type 2 diabetes that were tracked in 35 physician practices participating in the team-oriented, practice-based CME program. Early results from a four-practice pilot program showed that 75 percent of patients' blood pressure scores improved from baseline, and an average of 3.6 new patients participated in group sessions each month. 

  2. Association of California Nurse Leaders/CalNOC tracked data reported to the California Nurse Outcomes Coalition data repository before and after interventions in about 90 participating medical-surgical units in 32 hospitals to reduce falls and fall-related injuries, compared to 260 non-participating units in the same hospitals. Pre-post data analysis found mean change in falls and mean change in falls with injury were not significantly different between participating and non-participating units. While the falls per 1,000 patient days in participating units decreased slightly after the intervention, project researchers are trying to determine if the lack of a statistically meaningful difference is due to improved reporting, widespread attention to falls due to a JCAHO focus during the intervention period, or the interventions not having sufficient impact on a relatively rare event.
  3. When PFQ began, Catholic Healthcare Partners already had a system to report quality of care processes for treatment of heart failure patients via MIDAS, a proprietary data warehouse for hospital benchmarking. It collected data on ACE inhibitors prescribed at discharge, left ventricular function assessment, smoking cessation counseling, and appropriate discharge instructions. The PFQ project, called Heart Failure (HF) Guidelines Applied in Practice (GAP), aimed to attain a score for each of the four measures at or above 75 percent of all HF patients or the top 25th percentile in MIDAS, whichever was greater. It also set an organization-wide goal of reducing 30-day all-cause readmission rates for patients with an HF admission. About 18 months after implementing interventions in six hospitals, preliminary results indicate that patients under the care of HF advocates experienced a 41 percent drop in readmissions, and almost a doubling of the period between readmissions.  
  4. Child Health Corporation of America (CHCA)'s quality improvement strategies focused on several areas, including hospital patient safety, medication safety and pain management, and initiated many QI projects involving different subsets of CHCA member hospitals. One of the most successful projects involved an effort to reduce adverse drug events (ADEs) related to narcotics. Over an 18-month period, the 18 hospitals participating in this project showed a 49 percent decrease from 39.1 to 17.1 ADEs per 1,000 narcotic doses. Another successful project focused on reducing bloodstream infections by implementing best practices in 29 hospitals. The results showed 57 percent improvement in infection rates for 18 of the 29 hospitals, a drop in bloodstream infections from 6.9 to 4.8 per 1,000 line days for all 29 hospitals, and 88 percent compliance with IHI and CHCA-created "best practice" guidelines.
  5. The International Severity Information System (ISIS), whose PFQ project streamlined nursing facility documentation of patient care processes, tracked operational measures related to interventions and clinical care measures for pressure ulcers. Seven facilities that implemented interventions starting in April 2005 reduced the number of high-risk patients with pressure ulcers by 33 percent. Pressure ulcer prevalence in participating facility units dropped over the project period to 8.7 percent on average, compared to the national average of 14 percent, which remained flat over the life of the project. Facilities that implemented the interventions more completely, such as regularly submitting care process forms and using the reports in care planning meetings had better results—pressure ulcer prevalence of about 5-6 percent—than those that partially implemented the interventions.  
  6. Lehigh Valley Hospital and Health Network (LVHHN), which provided a package of educational interventions to physicians and patients to improve care of type 2 diabetes patients, monitored process of care measures and clinical lab scores for selected patients in participating primary care physician offices at baseline, six months and 12-months post-intervention. About 18 months after the start of the project, it reported improvements in the percent of physicians screening for glycosylated hemoglobin (HBA1c) and lipids (but not micro-albuminuria) in a timely manner relative to ADA guidelines. Patients also showed progress in adherence to recommended practices and statistically significant improvements in blood pressure, lipid levels, cholesterol, triglycerides and hemoglobin. 
  7. Physicians Micro Systems, Inc. (PMSI)/Medical University of South Carolina (MUSC) sought to improve adherence to clinical guidelines for more than 70 indicators in eight sets of medical conditions, including heart disease/stroke, diabetes, cancer screening, immunizations, respiratory disease, mental health and substance abuse, nutrition and obesity, and drug prescribing for the elderly. Participating practices all used PMSI's electronic medical record system, which made it easy to extract data and generate quarterly reports. MUSC staff and consultants provided educational services and support to physician practices on clinical guidelines in each area. Preliminary results indicate statistically significant improvements in the summary index measure for the percent of eligible targets met in the 78 indicators, rising from 33 percent at baseline (9/02) to 46 percent three years later. According to the project investigator, the results are not as large as they could have been if the project had focused on a smaller number of practices and fewer quality indicators. 
  8. Visiting Nurse Service of New York (VNSNY) worked with eight home health agencies from around the country on its first phase of quality improvement efforts, focused on care for diabetic patients.  Each agency submitted monthly data from chart reviews on clinical measures related to glycemic control, foot care, and medication management.  The proportion of people with diabetes receiving a comprehensive foot exam by a nurse within 10 days of admission to home care increased more than 50 percentage points over the course of the project.  Also, patients with blood pressure in their target range most or all of the time increased 30 percentage points, with similar increases in patients who received and an individualized glycemic control plan, foot care education and a review for medications with possible contraindications.  The second phase of the project, which focuses on reducing hospitalization in home care patients, has preliminary data suggesting a drop of 2.5 percentage points for the 70 home health agencies.

4. Effects of Projects Focused on Infrastructure and Learning

Among the 17 projects that were trying to improve clinical quality of care, three that focused on health care providers (AMA, JCAHO, RTI) and two that focused on purchasers (The Leapfrog Group and HealthFront) had goals that could not be measured quantitatively. As mentioned in Chapter III, only two of these five projects—the AMA and The Leapfrog Group— tried to formally evaluate their success, so we have limited ability to judge the effects of the other three projects. 

Of the three provider-based grants focused on infrastructure and learning, two involved major national organizations (AMA and JCAHO).  AMA's work to examine electronic transfer of data for performance measurement had, sponsors say, important lessons about the practical issues and challenges to data extracting exporting and validation.  With CMS and others calling for the introduction of performance measures for physicians in office-based practice, these findings have the potential to be very important.  JCAHO's work involved a survey of hospitals about their perceptions of the value of performance measures, as well as a comparison of self-abstracted data on performance measures with data abstracted by third parties. They found that the self-abstracted and third-party abstracted data is essentially similar, which may help build confidence that hospitals' own data is reliable enough to use in pay-for-performance systems. 

Among purchasers, The Leapfrog Group worked with purchasers in six markets to encourage use of quality information in selecting hospitals.  Though Leapfrog sought to evaluate the effects of these efforts, only three of its six pilot projects were implemented and evaluation results were available from only one of the pilots for this report. That pilot involved a differential patient co-payment to encourage use of hospitals meeting Leapfrog's quality and patient safety practices. Preliminary results show no effects on choice because physicians' admitting privileges appear to play a stronger role in influencing patients' hospital selection. Leapfrog continues to evaluate these efforts and says that it has gained valuable experience in establishing pay-for-performance programs.

There was no information on impacts of the projects led by RTI and HealthFront, although HealthFront reports that stakeholders in the two markets it targeted have been interested in the results from surveys of providers' perception of incentive and reward programs.


12. RE-AIM is a "systematic way for researchers, practitioners, and policy makers to evaluate health behavior interventions.  It can be used to estimate the potential impact of interventions on public health," according to its developers.  For more information, go to http://www.re-aim.org/index.html and Glasgow, et al., 1999.  AHRQCoPs Subcommittee on Dissemination and Impact also found the RE-AIM framework useful in examining the impact of three PFQ projects.

13. For example, in the RE-AIM framework adoption refers to the percentage and representativeness of the sites or providers that agree to participate.  The representativeness of the participants is important because the results cannot be generalized or may not be broadly replicable if those who participated are more motivated or ready to change than those who did not.  This is difficult to assess in the PFQ projects. Because these were applied research projects, virtually none of them randomly selected organizations to participate.  A few projects tried to compensate for this by randomly assigning those who agreed to participate to an experimental or control group, or to one or another intervention. A few stated that they tried not to recruit those who were innovative or best-in-class, but they were not able to verify this with any data. Thus, this analysis does not address adoption.

14. Partnerships for Quality. Fact Sheet. AHRQ Publication No. 04-P004, March 2004. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/qual/partqual.htm

15. Among the four projects with clinical practice or outcome goals whose results are not yet known (AAP, AHA-HRET, AMDA, NYS-DOH), one has indicated it expects positive impact, but implementation delays and problems with the other three indicate that they may not have as positive results to report as those in the eight projects with preliminary findings.


Return to Contents
Proceed to Next Section

 

The information on this page is archived and provided for reference purposes only.

 

AHRQ Advancing Excellence in Health Care