Skip Navigation Archive: U.S. Department of Health and Human Services www.hhs.gov
Archive: Agency for Healthcare Research Quality www.ahrq.gov
Archive print banner
Evaluation of AHRQ's Partnerships for Quality Program

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: https://info.ahrq.gov. Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to www.ahrq.gov for current information.

Chapter VI. Contribution of Partnerships and Other Key Factors to Project Success and Sustainability (continued)

C. Role of Other Key Factors in Project Success/Sustainability

While the PFQ projects all used some form of partnership to accelerate the translation of research into improved health care quality, safety, and security, they faced many challenges to changing professional and organizational behavior.  Below are the most significant factors that appear to have enabled or hindered progress in the PFQ projects, and how they tried to overcome these challenges.

1. PI Leadership

Many of the partners interviewed for this evaluation stressed the contribution of the leadership by the principal investigators (PIs) and others in the leadership team as a key factor in their perceived success in implementation and diffusion. The particular qualities of leadership differed from person to person, but they all functioned as champions in one way or another. Some partners mentioned the PIs' energy and enthusiasm for the project as a key factor in the success of the project, while others cited his or her expertise in the subject matter.  Several partners credited their projects' successes to the support and ideas provided by the lead organization staff, their willingness to work collaboratively with providers, and their flexibility in dealing with problems that emerged. In contrast, one project partner mentioned the PI's lack of organization as a detriment to greater success, another said turnover in leadership slowed the project's progress, and a third said that one of the partners didn't really play a strong leadership role, leading to failure to launch a pilot project in one site.

However, to succeed, PIs need more than a stellar record of research published in peer-reviewed journals. As the previous section stressed, PIs and their leadership teams must have experience in partnership management to structure and use them effectively.  PIs that had these skills, or could invest the time to develop them, appeared to be more effective in harnessing their partners' contributions towards the attainment of project goals.

2. Good Timing and a Supportive Environment

Some projects benefited from external developments and forces that lent their efforts greater relevance or urgency with the target organizations.  The bioterrorism preparedness projects had an initial advantage in this regard, since memories of terrorist and anthrax attacks in September and October 2001 were still fresh when the PFQ projects began in September 2002.  The Katrina and Rita hurricanes in the fall of 2005 represented important reminders of the need for the health care system to be prepared to deal with emergencies, and increased interest by partners in working with Altarum Institute's and Texas A&M University's projects.

As the drive to implement pay-for-performance and electronic health record systems gained momentum, driven by CMS and the Office of the National Coordinator for Health Information Technology, as well as large national health plans and employer purchaser groups, the PFQ projects that worked with providers to help them measure and report their performance against national standards also gained relevance. One PI said, "Our timing for the project was also right because the grant started just before pay-for-performance got big, and we had it up in time before the P4P angst started. At that time, our [physician] members were tired of the talking-head learning experience and were ready to do something in their practices."  Another PI affirmed this sentiment:  "People are cognizant of the IOM studies and realize that we're not doing as good a job as we should be, but then people don't know how they should be doing things differently. This project came in and offered to show the physicians how to do it."  Increased expectations for physicians to use electronic medical records had the same constructive effect. "It also helped that the practice sites knew that EMR was where all the big groups were headed. It helped to have a mix of a few small sites and few big organizations because that reinforced to the small sites that rather than being just another academic exercise, this was where the industry was going." Such forces help to overcome resistance to change, though they do not always succeed. Hospitals' resistance to being held accountable for performance outcomes blocked progress in several of the Leapfrog Group's pilot projects, for example.

Several projects' experiences reinforce the importance of picking the right health condition for focus.  AAP was glad it decided to focus on ADHD because "it was an easy sell—the interest was very high... the topic had a lot to do with it, so we did not have much of a problem with recruitment."  The long-term care projects' focus on pressure ulcers in LTC facilities, and primary care practices' focus on diabetes care benefited because these are conditions on which providers are more likely to be measured and reported in current or emerging public reporting systems.  

3. Ability to Overcome Provider Resource Constraints

To secure provider participation, and successfully implement their interventions, all projects needed to overcome common barriers confronting providers. Most health care organizations face the pressure of limited funds, time, staff, and other resources needed to test new approaches to quality improvement, patient safety, and emergency preparedness.  Even if they recognize its potential value, natural resistance to behavior change and uncertainty about the benefits of new ways of working can be powerful deterrents to adopting new practices. And even when change begins to take hold, staff turnover at all levels can affect the pace of progress. As the following quotes show, these issues presented enormous problems in nearly every project:

  • Time and Competing Priorities.  "Lack of time and money and an overwhelmed environment were the challenges that most hindered our progress... the practicing physicians are incredibly overwhelmed. People do not want to take on this kind of [work] because it will increase the workload..." "The competing priorities of the organizations were a huge barrier to trying to get anything done. They've got so many things people are telling them they've got to get done..." "Practices are just so busy, and even the highly motivated practices see this as an add-on to their daily routine."  "To some facilities, this just seemed like "another project" that would take a lot of time without being certain it would improve their quality measures." "At the end of the day, when someone is volunteering and there are multiple demands on their time, we can't dictate the progress they make. That's our biggest stumbling block—that we don'thave a command and control scenario."
  • Funding.  "[Although] the program was 'free' it required them to devote staff time to something that didn't have a guaranteedreward or positive outcome." "The business case is very difficult... there are many hospitals where even if they wanted to do this, they can't afford to do it upfront."  "While the pot of money at the top [for bioterrorism preparedness] looks big, by the time it gets to the states and the states divvy the money up to their regions, there isn't much left."
  • Turnover:  "An inhibiting factor is turnover at the senior leadership level. If you get turnover at the chief nursing officer or nurse manager level, you potentially have to start over, so that hinders us at the longitudinal level." "The turnovers are tough. The turnover at the ______ plan caused us to lose momentum, and even though [a project collaborator's move to another organization] was a blessing in disguise, the project lost time because of it." "In some cases, we would get all ramped up but then go back a month later and the person was gone." "At one hospital, the CEO left and a new person took over who didn't buy into the [program]..."
  • Speed and willingness to change.  "One of the challenges for all agencies... was getting the nurses to change what we wanted them to change at the speed that we wanted them to—having to continually get people to buy-in. "... different doctors went through the stages of change differently.  Some went through the stages easily and other took much longer. Some doctors tested us by giving us the toughest patients first so they could see what we did with them. Eventually, when they saw that we dealt with those patients well, they were persuaded to engage more."

Successful efforts to overcome provider resistance required flexibility and smart use of available resources.  For example, some projects modified their interventions to reduce barriers to participation, or gave providers the ability to adapt the intervention to their organizational culture or practice.  By design, some projects sought to provide more support than others, especially when their interventions required more significant change in organization policies or operations.  While most projects overcame the challenges associated with recruitment, they varied in their ability to provide sufficient flexibility and support to providers, which may have affected the degree of success in achieving project goals.

While some projects provided intensive training and support to target organizations to implement new quality measurement and improvement tools and techniques, other projects intentionally limited the amount of support they offered to providers after an initial training course, believing that more intensive follow-up support would not be sustainable after AHRQ grant funds ran out.  Examples of the latter model included projects run by NYSDOH and AHA/HRET, which provided target organizations with brief training courses or site visits, but had minimal follow-up, except for collecting data for evaluation purposes.  Preliminary results suggest that the first strategy—intensive follow-up support—was more successful in making or sustaining changes.  It may be that such support enabled participants to realize the benefits of the intervention more quickly, generating greater commitment. However, as final results are not yet known, this warrants further investigation.

Since staff turnover is inevitable, it is important to learn from those projects that found ways to minimize its impact on their interventions.  The most successful projects appeared to be those that worked with teams from organizations, rather than with one person.  That way, even if one of the team members left, the others were already on board and could train new staff. 

4. Effective Use of IT for Quality Measurement and Provider Feedback

Projects that made effective use of information technology to measure and motivate care process improvements had more measurable, and possibly better, progress in improving adherence to clinical guidelines or yielding higher scores on clinical outcome indicators. Eight projects (AAP, ACP, AMA, CHP, ISIS, Lehigh Valley, Physician Micro Systems/MUSC, and VNSNY) used IT-based measurement systems to give practitioners the measures and the tools to compare their own performance with others.

When the IT systems were working well, the ability to provide feedback on an immediate and regular basis gave providers "actionable information" that they could use in their day-to-day patient care and practice management, as well as strong motivation to improve if their scores were below national standards or those of their peers.  When combined with a rapid cycle quality improvement approach, such as IHI's learning collaboratives, projects could use the data to accelerate the testing and refinement of quality improvement methods.  For example, according to one interviewee, "there needs to be an IT system in place for data collection... You need to be able to do real-time data collection that will show you whether you are doing the right thing for patients."  A physician participating in one of the projects said that success was largely attributable to "the report that we receive quarterly 100-page pamphlets with all of the graphs." Projects that worked with EMR vendors, such as the PMSI/MUSC project, had an advantage in this regard, "Because of the way we've developed this network and they all use electronic records,there'sno work to get the data...."

Having available IT tools was not enough though, unless grantees could make effective use of them.  Logistical issues still present hurdles as the AMA project discovered.  "Physician practices had difficulty getting their data into an HL-7 format to get it transferred. Thatwas alessonon needing standards for data transfer..." Other projects found that just making tools available on a Web site doesn't guarantee people will access or use them, suggesting the importance of making web-based tools more interactive and a part of the learning/quality improvement cycle. 

5. Effective Leverage of Grant Resources

The fact that all projects were grant-funded sometimes worked for, and sometimes against, efforts to make progress.  On the positive side, the grant funds obviously provided financial support for many activities and infrastructure development that could not have been achieved without the grant. "We definitely would not have been able to pay for or support the coaches... or the hierarchical analysis without the AHRQ grant [and it] provided us with support to establish some things that we'll be able to continue," said one interviewee.  Another said, "By giving the chapters some money, we were providing them with a lot of infrastructure support."

Many PIs and their partners also said that the external deadlines and deliverables associated with the grant had a salutary effect. Several of them said that providers and partners, especially those participating in learning communities, had more incentive to implement quality strategies, if only to be able to report their progress at the next meeting or teleconference.  For example, said one PI, "Anytime you have a deadline, that's helpful.You had an element of peer pressure there as well [as motivation] to get things done in relation to this project."  One of their partners affirmed that "Having ___ hold you accountable with the conference calls [wasa motivation to do the work]. We had other meetings and conference calls that were held internally... which [also] helped the individual practices stay in line." Having deadlines, said another PI, "made us report back and provide data and say what we're doing at a level of scrutiny that pushed us forward... the external deadlines we had... [made us] continually focus."

On the negative side, the amount of grant funds needed to make large-scale change was limited in relation to the overall goal. Projects funded for clinical quality improvement projects had between $300,000 and $400,000 for each of the four years, while those conducting bioterrorism preparedness projects had just $100,000 for each of the four years, so it was unrealistic to expect the 20 projects to reach millions of people as the AHRQ RFA envisioned. 

In addition, the requirement to evaluate the project's impact led grantees to spend funds on research and data collection activities that reduced the amount available for project infrastructure or partnership management. Several PIs complained about the need to prepare and obtain Institutional Review Board approval for their data collection activities.18  For example, one said, "Dealing with IRBs was an enormous problem...in quality improvement work, we're being asked to adhere to standards of research, but we're not really doing research. This needs to be looked at in a big way." Others ran into resistance from providers in submitting data needed for the evaluation.  "The data collection was always a big problem... [it was a burden for practices and we haven't figured out how to make it easier,"  said one PI.

This suggests the need to revisit how best to document the impact of QI interventions while not running afoul of patient rights.  Whether or not grantees could have designed their work to avoid these problems is something AHRQ may want to consider in formulating future projects of this type. 

Return to Contents

D. Lessons on Elements of an Effective Partnership for Quality

If one is planning to use a partnership to accelerate the translation of evidence-based research into health care practice on a wide scale, there are a few things that appear to be necessary to the success of such an endeavor, with implications for other AHRQ efforts to fund projects involving partnerships. 

  • Partnership structure.  The composition, size, and form a partnership takes should fit the goals and scale of the project. If the goal is to make large-scale change, projects should seek intermediaries to help with provider recruitment, training, and ongoing support for quality improvement; and efforts should be made to build capacity of these intermediaries to continue this work on their own over the long-term.  Partnerships should try to recruit participants who are committed to the project and are well-connected to their peers.
  • Leadership.  National organizations and project directors that have strong credibility with, and influence on the target, should take the lead in partnerships. This affirms the importance of taking the PI's reputation and track record into account when reviewing grant applications. It also supports AHRQ's practice of allowing PFQ projects to travel with the PIs when they switch employers, or transfer to different sponsoring organizations. In the context of partnerships, though, leadership does not equate solely with a record of scholarship and peer-reviewed journal articles; it also means having the enthusiasm for this sort of work, as well as commitment to, and flexibility in working collaboratively with partners.
  • Partnership management skills.  Leaders need skills and experience in partnership management, and make a commitment to spend time on forging consensus, fostering regular communication, sharing lessons, and resolving problems at all partnership levels. Partnerships that involve all partners in decision making and staff at all levels in the target organizations in tailoring the intervention to their own organization may be more successful in building commitment and sustaining activities in the long-run.
  • Strategies to overcome provider constraints.  Partnerships should anticipate and prepare tools and strategies to address the needs and constraints of providers.  They should also decide in advance how much room to allow providers to adapt the intervention so that it fits each organization's culture, and can be adjusted to each provider's pace of change.
  • Effective use of data and IT.  Partnerships to improve quality should consider seriously how best to make effective use of IT and data collection to measure and motivate providers to make care process improvements in "real-time".  
  • Regular interaction.  Partnerships should organize regular opportunities for organizations and providers to talk or meet with each other, since the need to report progress, share successes, and learn what works and what does not appears to accelerate providers' progress.
  • Timing.  If at all possible, the initiative should be timed to take advantage of external demands on providers that make the intervention more relevant and responsive to those demands. 

This list mirrors most of the criteria that AHRQ set out in the RFA for applicants to the PFQ program, affirming to a large extent the assumptions and thinking that went into the program's initial development. When one looks at the qualifications and proposals of the grantees that were originally funded in 2002, most met the majority of these criteria.

Projects that met the PFQ applicant criteria closely and put into practice these elements of effective partnerships appear to be most successful in achieving their goals or those of the overall program. Projects that did not meet the criteria as well, or were not able to apply these elements of effectiveness, appear less successful. As a new program for AHRQ, PFQ represented a form of venture capital, and as with all such investments, one can expect a certain number of failures. Despite the fact that some projects did not succeed as much as program architects may have hoped, they too have the potential to shed insight into the challenges of doing this type of work.


18. It is unclear whether IRB approval was required by AHRQ or by the sponsoring institution for many of the PFQ grant projects.


Return to Contents
Proceed to Next Section

 

The information on this page is archived and provided for reference purposes only.

 

AHRQ Advancing Excellence in Health Care