Skip Navigation Archive: U.S. Department of Health and Human Services U.S. Department of Health and Human Services
Archive: Agency for Healthcare Research Quality
Archival print banner

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to for current information.

Comparative Health System Performance in Accelerating PCOR Dissemination (U19)

Technical Assistance Conference Call

Slide Presentation (Text Version)

On August 22, 2014, the Agency for Healthcare Research and Quality (AHRQ) held a technical assistance teleconference on the Agency's funding opportunity announcement, "Comparative Health System Performance in Accelerating PCOR Dissemination (U19)". This is the text version of the slide presentation. 

Select to access the PowerPoint® slides (263 KB).

Slide 1 

Comparative Health System Performance in Accelerating PCOR Dissemination (U19)

Technical Assistance Conference Call
August 22, 2014

Slide 2 

Conference Call Overview

  • Introductions.
  • Background.
  • Review of the U19 Request for Application.
  • Frequently asked questions.
  • Open Q&A.

Slide 3 

Brief Background

  • Patient-Centered Outcomes Research.
  • Systems.

Slide 4 

Patient-Centered Outcomes Research

  • AHRQ has a long history of supporting and building evidence-based healthcare.
  • Patient-Centered Outcomes Research (PCOR) is comparative clinical effectiveness research of the impact on health outcomes of two or more preventive, diagnostic, treatment, or healthcare delivery system approaches. PCOR produces not only clinical findings, but also evidence about the effectiveness of different systems for delivering care.
    • Section 6301(a) of the Patient Protection and Affordable Care Act.

Slide 5 


  • The projects will be funded through the Patient Centered Outcomes Research (PCOR) Trust Fund.
  • The Affordable Care Act (P.L. 111-148) authorizes AHRQ to help disseminate research findings from the Patient-Centered Outcomes Research Institute (PCORI) and other government-funded comparative effectiveness research:
    • (Public Law 111-148 and its amending of Section 937 of the Public Health Service Act, 26 U.S.C. 9511(d)). 
  • Section 937(a) calls for AHRQ to "create informational tools that organize and disseminate research findings for physicians, health care providers, patients, payers, and policy makers."
  • The foundational data and tools developed by these projects will be necessary inputs into any future endeavors to disseminate PCORI findings.

Slide 6 

Two ends of the spectrum in AHRQ's recent PCOR Portfolio

  • Accelerating the Dissemination and Implementation of PCOR Findings into Primary Care Practice (RFA-HS-14-008)
    • focuses on improving PCOR dissemination within individual or small physician practices. [closed July 3]
  • Comparative Health System Performance in Accelerating PCOR Dissemination (RFA-HS-14-011)
    • focuses on the impact of large, growing, and evolving systems on PCOR dissemination and system performance. [closes Oct 17]

Slide 7 

Systems Background

  • Physicians increasingly work in delivery systems with new types of complex and virtual arrangements, incentives, and ownership structures that we do not yet fully understand.
  • Within these systems, PCOR findings are only valuable if they are effectively used by organizations and by the physicians and staff within these organizations.
  • It is not clear how to effectively target PCOR and other comparative effectiveness research dissemination efforts if we do not understand how these new delivery systems are disseminating PCOR.

Slide 8 

Purpose of FOA

AHRQ seeks to award cooperative agreements to up to three Centers of Excellence.  Each Center will carry out two related sets of activities:

  1. Create a Data Core that identifies and classifies delivery systems and includes data on their characteristics and their performance.
  2. Conduct up to 6 major projects over the 5 year project period, using data from the Data Core, that compare the performance of different types of health care systems.

These projects will:

  • Seek to identify, characterize, classify, and track the proliferating forms of health care systems.
  • Assess the extent to which they use PCOR evidence.
  • Assess the association between use of PCOR evidence and quality and cost of care.
  • Seek to understand the characteristics of high performing systems.
  • Disseminate this information to system leadership, patients, policymakers, and payors.

Slide 9 

Definitions of important terms

  • Eligible Systems.
  • Eligible System Comparisons.

Slide 10 

Eligible Systems to Study

  • Eligible systems will include physicians and hospitals, formally or informally connected to each other, and may also include formal or informal connections with other types of providers such as post-acute care facilities and home health agencies.
  • Formal connections between providers can occur via ownership or via contract.  For example, a system based on ownership might include:
    • One or more hospitals and the physicians whom they employ.
    • A health insurance company and the physicians whom it employs.
    • A medical group and the hospital or hospitals it owns.
    • One or more hospitals, the physicians they employ, and the rehabilitation facilities owned by the hospitals.

Slide 11 

System Examples

  • A system based on contracts might include, for example:
    • One or more hospitals and one or more physician-hospital organizations (PHOs) with which the hospitals contract.
    • A medical group or independent practice association (IPA) and the hospital or hospitals with which the medical group or IPA contracts to operate as an accountable care organization (ACO).
  • Informal connections among providers may also create "virtual" systems of care.  For example, informal systems might include:
    • A hospital and its voluntary medical staff (physicians who admit patients to the hospital but are not employed by the hospital).
    • A medical group or IPA that does not formally contract with a hospital, but that cooperates with the hospital through informal agreements on ways to improve patients' care.
  • The examples given above are meant to be illustrative, not fully inclusive; responses to this FOA may describe other system forms that are proposed for study.

Slide 12 

Eligible System Comparisons

  • Applications will be viewed as responsive to this FOA to the extent that they include comparative analyses of system performance that include:
    • Types of systems.
    • Incentives that the systems receive.
    • The integration of PCOR evidence into systems' processes of care.
    • The relationship between use of PCOR evidence and the quality and costs of care the systems provide.
    • The environment in which systems are located. 
  • Applications that focus on only one of these areas—for example, applications that compare different pay-for-performance programs without regard to delivery systems, or applications that compare different processes for reducing readmissions, without regard to the delivery systems within which these processes are embedded—will not be considered responsive to this FOA and will not undergo peer review.

Slide 13 

Eligible System Comparisons (2)

  • Applicants should explain how they will assess system performance across a broad spectrum of care.
  • Applications that focus only on a narrow area of care – such as outcomes after joint replacement surgery – will not be considered responsive to this FOA.
  • However, comparative performance on narrow areas of care can be eligible under this FOA if the comparisons are included in a broader assessment of system performance. 
    • For example, some systems might have lower overall costs of care for their patients because they provide better orthopedic care, or better cardiac care. 

Slide 14 

Eligible System Comparisons (3)

  • Comparing the performance of systems may include, for example, some or all of the following types of projects:
    • Comparing the performance of individual delivery systems to other delivery systems:
      • Within the same class of delivery system.
      • Across different classes of delivery system.
    • Comparing the performance of one class of delivery system.
    • To one or more other classes of delivery system comparing the change in performance over time of an individual delivery system or class of delivery systems.

Slide 15 

Guidance for Applicants: TEP

  • Each PI will participate in the Technical Expert Panel (TEP) meetings.  The TEP, a panel of outside experts, will meet once a year in addition to telephone conferences and electronic exchanges. 
  • For budget purposes, applicants should assume that the TEP will meet in person once a year at AHRQ, with 4 people traveled.
  • In response to the TEP meetings, the PI will actively participate in the formulation of plans to promote generalizability across Centers of Excellence (harmonization of measures (not necessarily of data) for comparing systems). 
    • Grantees should expect to cooperate in efforts across Centers and with AHRQ to develop and harmonize measures for comparing systems.
  • Awardees will refine and revise, as necessary, selected methodologies in accord with plans developed collaboratively with the TEP.
  • The grantees should be prepared early on in their grant to present methods and early findings to the TEP, and to publish early findings.

Slide 16 

Guidance for Applicants: Coordinating Center

  • AHRQ plans to fund a Coordinating Center under separate contract to assist in dissemination and data work.
  • The PI and the Coordinating Center will work collaboratively to build a taxonomy and systems compendium or compendia of health care delivery systems found across the country.
  • AHRQ will share its own potential data and results with the Coordinating Center's compendium, and the awardee is expected to share data results.
  • For purposes of the collaborative research, the systems compendium will list systems by name and include identifiers – e.g., tax identification numbers (TINs), hospital identifier (AHA id, Medicare provider number, etc.),  and physicians' national provider identifiers (NPIs) as appropriate - with outcome measures and  performance of the system listed.
  • Using the systems compendium, both AHRQ and the PI will collaborate with the Coordinating Center so that  the Coordinating Center can develop short Policy Briefs and Data Briefs on comparing systems, and can develop PCOR dissemination initiatives.

Slide 17 

FOA Basics

  • AHRQ is utilizing the U19 mechanism.
  • AHRQ anticipates making up to 3 awards.
  • Grants are limited to $3.5 million total costs per year.
  • The project period may not exceed 5 years.
  • "No cost extensions" will not be automatic.

Slide 18 

Eligible Organizations

  • You may submit an application(s) if your institution/organization is a (an):
  • Public or non-profit private institution, such as a university, college, or a faith-based or community-based organization;
  • For-profit private institution;
  • Units of local or State government;
  • Eligible agency of the Federal government.
  • Indian/Native American Tribal Government (Federally recognized) Indian/Native American Tribal Government (Other than Federally recognized); Indian/Native American Tribally Designated Organization.

Slide 19 

Principal Investigator/Project Leads

  • The PD/PI is required to devote at least 15% annual effort in each year of the project.
  • Also, the leader of the Data Core and the leaders of each project must each devote at least 15% annual effort in each year of the project.
  • Only one PD/PI allowed.

Slide 20 

The Application

  • Page Limitations
    • Data Core:12 pages.
    • Inter-related Projects: 5 pages per project.
  • A planned project timeline must be included, identifying timing of major milestones such as planned release of early findings.

Slide 21 

Review Criteria (I) Significance

  • The review criteria provide an outline of both what AHRQ is seeking and the questions peer reviewers will be asked to consider.
  • The Significance of the application will be based upon:
    • Does the project address an important problem or a critical barrier to progress in understanding and identifying new types of emerging delivery systems? 
    • Will the proposed project lead to a better understanding of current and emerging health care delivery system models in order to better target PCOR dissemination? 
    • If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved?
    • What will be the effect of these studies on improving PCOR dissemination and improved uptake of CER findings?

Slide 22 

Review Criteria (II) Innovation

  • Is the project original and innovative? Does it:
    • Seek to improve our understanding of how to compare the performance of different types of health care systems?
    • Assess the extent to which these systems use PCOR evidence?
    • Assess the quality and cost of the care systems provide?
    • Seek to understand the characteristics of high-performing systems?
    • Propose innovative methods for disseminating PCOR evidence within health care systems, or address an innovative hypothesis or critical barrier to progress in PCOR dissemination?
    • Develop or employ novel concepts, approaches or methodologies for this area, such as for dealing with selection issues or issues of assigning  patients to systems?

Slide 23 

Review Criteria (III) Approach

  • Is a planned project timeline included, and does the timeline include availability of early findings?
  • Are the health care systems comparisons proposed in the application well justified?
  • Does the applicant propose an appropriate budget for the Data Core activities?
  • Does the application include a plan to share analytic results, tax identification numbers (TINs), and physicians' national provider identifiers (NPIs) with the Coordinating Center for the systems compendium? 
  • Does the application include a well-developed plan for acquiring data?
  • Does the Data Core include methodological work on how to statistically compare systems in terms of their characteristics and performance?
  • Do primary data collection efforts have the appropriate methods?

Slide 24 

Review Criteria (IV)
Approach Cont’d

  • Do proposed inter-related projects describe systems to be studied, system processes, types of incentives, and what environments and markets speed up the diffusion of PCOR findings into practice to produce patient-centered outcomes?
  • How does the applicant plan to assess the impact of systems on PCOR findings?
  • Are proposed projects well designed to have a major impact on health care in the United States? 
  • Has the applicant defined and justified the systems and geographic area to be studied? 
  • Do proposed projects compare the outcomes of different types of systems, processes, incentives and environments? 
  • Does the applicant acknowledge potential problem areas and consider alternative strategies?
  • Are benchmarks for success presented?

Slide 25 

Important Dates

  • Letters of intent are due September 5, 2014.
  • Application due date is October 17, 2014.
  • Peer review is estimated to be January 2015.
  • Grants start date is estimated to be April 2015.

Slide 26 

Letter of Intent

  • Highly encouraged, non-binding, not required.
  • Letter of intent should include:
    • Number and title of this funding opportunity.
    • Descriptive title of proposed activity.
    • Name, address, and telephone number of the PD/PI.
    • Names and institutions of other key personnel.
    • Participating institution(s).
  • The letter of intent can be sent electronically to:

Slide 27 

Additional Help

Slide 28 


  • Q: Does each individual project need to examine PCOR dissemination?
  • A: The goal of the combination of interrelated projects is to address the following question: What kind of systems, using what types of processes, with what types of incentives, and in what kinds of environments, speed up the diffusion of PCOR findings into practice to produce the best patient-centered outcomes? The applicant should identify how each individual project contributes to this goal.

Slide 29 


  • Q: Does the PI need to lead an individual project?
  • A: No, the PI must just have at least 15% time on the overall grant.
  • Q: If the PI also leads the Data Core or a project, will the PI need at least 30% time
  • A: Yes.

Slide 30 


  • Q: Must all the projects address a common theme as in a P01?
  • A: The FOA has a targeted theme: what are the characteristics of systems that perform well, particularly with respect to PCOR dissemination.
  • The projects need to be inter-related with respect to this main PCOR theme, and the application will need to be clear about this inter-relationship.

Slide 31 


  • Q: Structure. Should a Center focus more on depth (examining what happens within systems), or more on breadth (what is happening across the country over time)?
  • A: We will entertain both types of proposals. The applicant should indicate and defend the choice made, and how their application can best contribute to achieving the overall goals of the solicitation.

Slide 32 


  • Q: Are we allowed to examine interventions?
  • A: Yes, interventions already funded can be studied as part of the overall project. See discussion earlier on what is and is not in scope.

Slide 33 


  • Q: FOA specifies page limits for the data core (12) and each of the inter-related projects (5), but not for the overall research strategy.The U19 Activity Code does not appear in the NIH page limits table. Can the overall research strategy be described separately, or is it expected to be included in the data core section?
  • A: An overall research strategy can be put in the Research Plan section in the "Overall Component." There is no page limitation for that.
    • The PHS 398 says, "Prepare a succinct Research Plan…Sections 4-15 of the Research Plan have no maximum allowable pages, but should also be succinct."

Slide 34 

Open Forum

  • The operator will assist in queuing questions from participants.
  • If time does not allow for all questions to be answered, please submit your questions via Email after the call.
Page last reviewed September 2014
Internet Citation: Comparative Health System Performance in Accelerating PCOR Dissemination (U19): Technical Assistance Conference Call . September 2014. Agency for Healthcare Research and Quality, Rockville, MD.


The information on this page is archived and provided for reference purposes only.


AHRQ Advancing Excellence in Health Care