Skip Navigation Archive: U.S. Department of Health and Human Services www.hhs.gov
Archive: Agency for Healthcare Research Quality www.ahrq.gov
Archive print banner
Evaluation of AHRQ's Partnerships for Quality Program

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: https://info.ahrq.gov. Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to www.ahrq.gov for current information.

Chapter I. Background on the PFQ Program and Evaluation Goals

The Partnerships for Quality (PFQ) program sponsored by the Agency for Healthcare Research and Quality (AHRQ) aimed to accelerate the translation of research findings into practice on a broad scale through public-private partnerships led by organizations well-positioned to reach end users.  PFQ was one of AHRQ's earliest efforts to structure work in ways designed to support this goal.  As a result, AHRQ is very interested in understanding what can be learned from the experience.  To support this interest, AHRQ contracted with Mathematica Policy Research Inc. (MPR) in the last few years of the program to evaluate PFQ and the lessons it might have for future efforts in translation. 

In this first chapter of the final evaluation report, we review:  1) why partnerships are important to AHRQ's goals, 2) the origins and purpose of PFQ, 3) the grantee solicitation process and grantees selected, 4) the infrastructure AHRQ created to promote and oversee the success of the program, and 5) the key evaluation objectives.   

A. Relevance of Partnerships to AHRQ Goals

The Agency for Healthcare Research and Quality (AHRQ) is increasingly focused on improving health care delivery and outcomes (Gray et al. 2003; Clancy 2004b).  In its efforts to improve quality, AHRQ engages in four types of work: research to support evidence-based decision making; use of data to drive quality; accelerating the pace of quality improvement; and improving the infrastructure for quality health care (for example, informatics).  (Clancy 2004a). AHRQ also views itself as the "science partner" to the Centers for Medicare and Medicaid Services and the states with respect to quality improvement. Collaboration is essential, given what AHRQ's director Dr. Carolyn Clancy has termed the "Quality Challenge," as reflected in the gap between current practices and what we know from research to be effective (Clancy 2005). This is what commonly is referred to as the challenge of "translating research to practice."

A critical strategy used by AHRQ to reduce the gap is to accelerate the pace of quality improvement through partnerships with public and private sector organizations that can move research on effective care into practice across the health care system.  Through these partnerships, AHRQ seeks to encourage the adoption of practices that research has shown to be effective. Examples of such partnerships include programs such as the Primary Care-Based Research Network, the ACTION program (formerly the Integrated Delivery System Research Network), the Put Prevention into Practice program, and the Partnerships for Quality program, which is the focus of this study.  Through these and other programs, AHRQ seeks to strengthen its ties to organizations that are well-positioned to reach providers and other important parties able to influence health care delivery.

Research suggests that partnerships such as those AHRQ is investing in are critical to enhancing the use of evidence-based practices (Greenhalgh et al. 2004).  For example, the diffusion of effective practices is more likely to occur if, among other things, it has the support of early adopters (opinion leaders receptive to change and well-integrated into the appropriate networks) (Berwick 2003).  If early adopters make their practices observable and gain the trust of relevant networks that are perceived as subscribing to similar values, further diffusion is much more likely to occur.  Thus, involving key leaders who are respected in health care or influential in its practice is vital to encouraging practice changes that improve health care delivery.

Return to Contents

B. Origins and Purpose of the PFQ

The process through which the Partnerships for Quality (PFQ) program was developed involved many people, including some no longer with the agency.  Through the solicitation, AHRQ was seeking to move beyond its original efforts at translation to reach a broader set of providers and others who were well-positioned organizationally to effectively translate research to practice. While AHRQ had previously attempted some work of this kind through the Translating Research into Practice Programs (TRIP I and II) and PFQ could be considered "TRIP III," staff also viewed the two sets of programs as distinct.  

As some characterized it, the TRIP program begun in 1999 was more about small-scale researcher-driven studies that worked with health care organizations to determine which techniques led to effective use of research in the delivery of care.  PFQ, on the other hand, aimed to encourage change in practice on a broad scale so that care was more consistent with emerging research evidence.  One AHRQ staff member, for example, said that while TRIP was trying to translate research into practice, TRIP ended up funding rigorous studies on how to change outcomes in well-defined populations and didn't have the reach intended by PFQ, which was meant to be broader to include the next generation. PFQ could reinforce, for example, ongoing partnerships between AHRQ and groups like the American Medical Association (AMA) and others that might be key to encouraging adoption—but unlikely to apply for a grant—and hence would not otherwise have a way to work closely with AHRQ on translation.

At the time the PFQ program was developed, AHRQ's strategy was evolving from one that supported quality improvement by funding the production of knowledge to funding promotion of broader use of knowledge.  The development of the PFQ initiative was indicative of a change in culture; the agency saw PFQ as the beginning of a series of projects with demonstrably broad impacts through which the agency could show "look, we are touching America," as one former AHRQ executive characterized it.  Through the PFQ program, AHRQ hoped to find out what could be accomplished and how sustainable it could be after the grants ended.

Top leaders of the agency (especially Lisa Simpson, who then was deputy and Carolyn Clancy who now is the director) say they conceived of the idea for the program and developed a one-page summary of it that reportedly was approved at AHRQ's Executive Management Meeting (EMM) (AHRQ's senior management group).  Staff members were then tasked to develop the concept into a Request for Application (RFA).  A senior staff member (Elinor Walker, since retired) was assigned to write the solicitation, working with a committee of AHRQ staff set up for the process.  Staff involved in the effort said that designing the solicitation was challenging because the goals of the program were so ambitious in relation to the limited funds available for it.

The final RFA resulted from an iterative process between AHRQ leadership and the RFA development committee. One AHRQ senior official described the RFA development process as difficult and contentious.  Though details reported by participants in the process are now somewhat vague and inconsistent, we understand that AHRQ leadership and the committee had to grapple with competing views on a range of issues including: who to target with the solicitation (traditional researchers versus others), whether to allow the substantive focus to vary each year based on emerging research results from AHRQ or elsewhere (versus maintain a single focus over the years), whether to focus on clinical concerns only or broader strategies for quality improvement, and how to balance the desire for nontraditional grantees who had broad reach with concerns that such grantees were not used to working under a grant mechanism that held them accountable for the funds and could have limited experience with evaluating their projects. AHRQ leadership wanted internal grantee evaluations that might help the agency show that its findings were reaching or being adopted by health care providers nationwide—information that would be invaluable in gaining support for funding agency programs. 

As ultimately released, the solicitation for PFQ applications was designed to encourage applicants beyond the usual academic institutions the agency had historically funded. AHRQ wanted to fund "change agents" that not only possessed the evidence-based knowledge to improve care but could also create the partnerships and had the capacity to influence changes in health care organization and delivery. The agency's desire to fund a different kind of grantee, these "change agents," required reworking the usual processes by which grantees were solicited, reviewed, and chosen. AHRQ barred universities from serving as grantees, though researchers affiliated with universities might be involved, even as project investigators.

Our ability to describe the origins of the PFQ program (or the decision process on awards) in more strategic terms is limited by the fact that many people who developed PFQ are no longer with the agency and many key decisions on strategy either are undocumented or not retrievable for the evaluation.  Though we were able to interview several current or former staff involved in the program, these interviews did not take place until several years after the program was initiated.  By then, some details were forgotten and some perceptions modified by more recent events.  

Staff turnover made it hard to cleave to the original PFQ vision.  Current PFQ experience needs to be understood in this context as do past pressures to address other priorities after the grants were awarded.  Without strong guidance from leadership, AHRQ's ability to ensure the original program vision and concept into the day-to-day work of program implementation was hampered, as discussed later in the report. 

Return to Contents

C. Grantee Solicitation and Awards

1. Solicitation Process

Because PFQ reflected a new program strategy for AHRQ, it required changes in the usual way grants are reviewed.  To facilitate PFQ's agenda, the agency had to ensure that 1) the RFA was different from previous solicitations, encouraging more health system leaders to apply than in the past, 2) the review panel maintained a good balance of academics and people with operational experience with health care delivery, and 3) the funding committee balanced both rigor and relevance in its funding decisions.  

The PFQ request for applications, released on May 10, 2002, sought applicants for cooperative agreements1 to conduct projects designed to "accelerate the pace with which research findings are translated into improved quality of care and the health system's ability to deliver that care."2

The solicitation encouraged applicants with the capacity to influence health care organization and delivery and evaluate the impact of their efforts, such as health care professional organizations, accrediting agencies, practice networks, employer coalitions, and health insurers. Academic institutions could be one of the partners, but not the grantee. The multi-year projects had to:

  • Identify high-priority areas that are important to core audiences and for which evidence-based findings can guide improvements.
  • Translate, disseminate, and implement evidence-based findings, with a preference for those supported by AHRQ research.
  • Annually update opportunities for collaboration and efforts to respond to issues on security, safety, quality, effectiveness or outcomes of care.
  • Estimate the impact of implementation efforts on policies, processes, or outcomes and stakeholders.
  • Facilitate AHRQ's understanding of research needs as perceived by diverse stakeholders relevant to the PFQ.

The solicitation described a program structure that envisioned an initial planning phase of one year for each grant and a second phase of up to three additional years of funding for grantees able to show potential.  By the end of the sixth month in the second phase, grantees would have to demonstrate that the aims would be accomplished if funding continued.  The solicitation also encouraged shorter projects with more limited objectives.  Budgets for the initial year were not to exceed $100,000, with subsequent annual funding potentially two to four times that amount. Funding for the second phase would depend on what had been achieved in the initial phase. Each project was to include an evaluation, as well as progress reports, at stipulated intervals.  In a late modification, AHRQ decided to expand the focus of PFQ to include projects relating to bioterrorism. In contrast to the original grants, which were made with AHRQ's direct funds, bioterrorism grants were funded with money AHRQ received from other parts of the Department of Health and Human Services to address bioterrorism readiness needs. The amount of the bioterrorism grant awards after the first year (about $100,000 per year) was considerably lower than the other awards under the PFQ program (about $300,000 to 400,000 per year).

We have little information on the selection process. Our interviews with staff suggest that AHRQ succeeded only partially in its efforts to recruit a more diverse review panel than was typical.  However, there was enough diversity of the panel to create some discomfort among those more experienced in the traditional review panel process.  For example, one participant told us:

There were people that felt that the reviewers were too researchy. I think I felt uncomfortable during a lot of the review because there was a lot of conflict, a lot of inconsistency. For some of the reviewers, the whole emphasis of the review would be on the research. And for some of the others, research wasn't sufficient. I didn't feel that a great many of them gave adequate attention to all the aspects—the feasibility, the likely value of the program, the evaluation. My feeling is that unless you've got a decent evaluation, you aren't going to learn much that you can use. I was kind of uncomfortable. But again, we didn't have a mode.

The panel had to have some researchers and some doers. The researchers didn't have a lot of meat to chew on and I was uncomfortable with what the doers were really bringing to the table. There were a lot of arguments and in each case, you weren't sure who to believe because you couldn't be sure what they were basing their comments on.  It's not that researchers don't have disagreements like that, but it's generally clear what they are basing their arguments on.

Not surprisingly, differences of opinion carried over into how participants on the panel viewed the ultimate decisions on awards.  Some staff told us with concern that AHRQ's final decision on PFQ awards did not strictly follow the ranked technical scores.  Some said that the review summary did not reflect the panel's richer views. AHRQ leaders, however, say that adjustments between proposals' ranking based on technical scores and actual awards are now routine to achieve a balance in work across topic areas.  

Another factor that complicated the grant selection process was that AHRQ planned to allocate funds to PFQ's overall budget for projects focused on children's mental health from a dedicated source.  Even though several applications that planned to focus on this area scored well, available funds were insufficient to fund all of them.  They were therefore "skipped" in order to fund projects focused on a broader set of health issues and conditions. 

Establishing an appropriate set of reviewers for grants as path-breaking for the agency as those envisioned under PFQ must have been challenging.  As one interviewee noted, awards needed to balance rigor against relevance, with an applicant pool other than "the usual suspects." Balancing traditional grant reviewers that focus on the rigor of design with other reviewers looking more at operational practicality probably was not easy.  We heard, for example, one AHRQ staffer say that some of the latter were not "objective" whereas another felt the panel didn't have enough experience with quality improvement.  It is unfortunate that detailed documentation of the review process is not available as it could have helped us provide more concrete feedback to the agency on lessons for future reviews of this sort.

2. PFQ Grantees

AHRQ spent about $20.5 million on PFQ grants over the program's life, of which about $17.6 million came from AHRQ appropriations and about $3 million from other departmental funding (Table I.1).  Twenty-two grants were made originally, with 20 remaining after the first year.  One of the 22 withdrew from the program before it received funding and another grant was not renewed after the first year.3 Grantees received an initial award and then up to three additional annual awards over the remaining period of the program.  The initial grants were awarded in late September 2002, with federal FY 2002 funds used to support work in federal FY 2003. The final fourth-year grants were made in September 2005 with an end date of September 2006. Of the 20 multi-year grants, 14 had an end date of September 30, 2006, although some of these grantees have applied for or received no-cost extensions so their work will continue into next fiscal year. 

Table I.2 lists the 21 grantee organizations, the principal investigators affiliated with each grant, the total award, and predicted end date and status as of September 2006.  Most grantees ultimately received the full four years of funding though funds were dispersed on an annual basis based on renewal application.  A few were for shorter periods of time, either by design or because problems arose. Since one of the original 21 grants was terminated after the first year, this evaluation focuses on the 20 grant projects that continued for more than a year.  We defer describing the characteristics and focus of these 20 grantees until Chapter III.

Return to Contents

D. Program Infrastructure and Oversight

AHRQ executives said that there was not a lot of discussion in advance of PFQ on how the program would be administered.  Establishing an administrative infrastructure was further complicated because several of the staff involved most closely with the program in its formation would soon be retiring or were otherwise unavailable. 

The PFQ infrastructure that was ultimately established appears to be a blend of the way AHRQ traditionally oversees grants with some program-wide elements designed to encourage synergy across grants on issues of mutual interest.4  This infrastructure relied on internal AHRQ staff and was not heavily resourced.  The basic elements of the infrastructure are as follows.

1. Organizational position of PFQ within AHRQ

PFQ was housed within one of AHRQs main operational centers—the Center for Primary Care, Prevention, and Clinical Partnerships (CP-3).  That was at least in part because Charlotte Mullican, who headed the program and monitored several grants, was located in that center, as were two project officers for six additional PFQ grants. 

Seven additional project officers who oversaw individual PFQ grants came from a variety of centers within AHRQ. PFQ was one of the first programs in the agency, in addition to TRIP I and II, to draw its project officers from across the agency (rather than from a single center), reflecting the scope of the program.  Individual project officers appear to have been assigned by dividing grants across AHRQ's line centers based on the grant focus.  Specific project officers were assigned by center directors based on availability of appropriate project officer staff. This resulted in a matrix form of organization in which individual project officers had lead responsibility for individual grants, while the PFQ program director managed program-wide meetings and tasks that would benefit from consistent efforts across grantees.  Early in the program, the program director had weekly meetings with project officers to discuss common elements of the program and issues of mutual concern (for example, grantee reporting requirements) though such meetings ended well before our evaluation began. 

In interviews, project officers conveyed different approaches to their oversight tasks.  From our perspective, there appear to be two different strategies taken by project officers. The first, typically preferred by project officers with a strong substantive interest in a given topic area, was to work closely with their grantees to help form linkages with others involved in the same issue. The second was what can be viewed as a more generic oversight role that focused on overseeing adherence with grant requirements rather than seeking involvement in the substance of the work. Project officers pursuing the first strategy typically focused more on work with individual grantees rather than program-wide activities, though they might do both.  Regardless of strategy, the amount of time spent by project officers on oversight varied substantially based on their interests and competing work assignments.

2. Program-wide Structure and Elements

AHRQ desired to encourage a program-wide focus with communication across grantees. The infrastructure to accomplish this included 1) periodic meetings of all grantees serving as a "Council of Partners;" and 2) a Web site where materials could be placed to foster communication. The concept behind the AHRQ Council of Partmers (AHRQCoPs) was not well-developed in the original RFA, though grantees were asked to include funds to attend an annual meeting.

AHRQ leadership appears to have left the decision on how to form AHRQCoPs to staff who, we were told, decided to model it on the structure used for the Translating Research into Practice (TRIP) grants.  At the initial AHRQ CoPs meeting, grantees were asked to elect leadership and approve a charter.  The intent was that AHRQCoPs was to be grantee run with AHRQ support.  Early meetings involved grantee presentations. Later on, the group divided into five subcommittees perceived to reflect the main challenges shared across all grantees:  1) science and partnership, 2) evaluation, 3) implementation, 4) dissemination and impact, and 5) sustainability.  Each subcommittee took responsibility for structuring one of the semi-annual COP meetings and set an agenda that addressed each subcommittee's area of interest (for example, implementation).  The meetings included a combination of speakers and time for subcommittee work. Later on, participants on AHRQCoPs suggested that they work together on a journal supplement that would complement their work by documenting what had been learned about their experience. This supplement was under active development at the end of the program.  The decision to focus on subgroups by cross-cutting challenge rather than substantive focus areas of the grantees was made after some debate among the program director and project officers.

In Chapter V, we provide additional details on the way AHRQCoPs functioned and how AHRQ staff and grantees viewed it as contributing to the success of their individual grants and the program as a whole.

Return to Contents

E. Evaluation Objectives

PFQ is a complex program involving a multiplicity of organizations and substantive foci. AHRQ asked that the evaluation not just document the richness of the program, but sort through the experience of diverse grantees to answer questions of interest to AHRQ.  These questions are:

  1. What impact did PFQ project activities have on improved health care quality processes and outcomes, and on the dissemination of effective quality improvement methods? In other words, how effective were the projects in accomplishing what they proposed and what AHRQ funded?
  2. Did PFQ generate partnerships and infrastructure important to sustaining change? How did the partnerships and networks created through the PFQ projects contribute to the project outcomes?
  3. How adequate was AHRQ's support and oversight of the program? How well did the agency support the projects and generate synergy and collaboration across projects?
  4. What contribution did PFQ make towards AHRQ's strategic goals, both through the individual projects and the program-wide activities?  

In addition, AHRQ leadership expected that the evaluation would inform internal management and operations of programs similar to PFQ.  For example, the results of the evaluation could inform the development of future RFAs and their review, funding processes for projects similar to PFQ, appropriate leadership structures for AHRQ programs that are cross-center versus those owned by a single center, and the roles and responsibilities of project officers in overseeing and documenting impact of grantee projects. 


1. Agencies signing cooperative agreements with AHRQ are not "grantees" in the traditional sense, since the cooperative agreement connotes a more collaborative relationship. "Cooperative activities are intended to strengthen individual projects and at the same time generate collaboration across projects." However, cooperative agreements are a type of grant and in practice, lead agencies were referred to as grantees, so we use this term hereafter. 

2. AHRQ Partnerships for Quality, Request for Applications (RFA): HS-02-010, Release Date:  May 10, 2002. http://grants1.nih.gov/grants/guide/rfa-files/RFA-HS-02-010.html

3. The American Board of Family Medicine was approved for work with NCQA to incorporate validated quality measures into recertification requirements for family physicians but the application was withdrawn before funding. In addition, the Pacific Business Group on Health received funding for one year before mutually agreeing with AHRQ to terminate due to its inability to obtain CMS data that was needed to implement its project.

4. Staff told us these were modeled after the formal councils that were part of AHRQ's ongoing work with the Translating Research into Practice program in Phase I and II.


Return to Contents
Proceed to Next Section

 

The information on this page is archived and provided for reference purposes only.

 

AHRQ Advancing Excellence in Health Care