Skip Navigation Archive: U.S. Department of Health and Human Services U.S. Department of Health and Human Services
Archive: Agency for Healthcare Research Quality
Archival print banner

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to for current information.

Comparative Health System Performance in Accelerating PCOR Dissemination (U19)

Technical Assistance Conference Call Transcript

Transcript of a technical assistance conference call related to Funding Opportunity Announcement (FOA), "Comparative Health System Performance in Accelerating PCOR Dissemination (U19)".

August 22, 2014, 2:00 pm ET

Please note: The FOA should be considered the source document for applications.

Coordinator: Welcome and thank you for standing by. At this time all participants are on a listen only mode until the question and answer session has begun. At that time if you would like to ask a question please press star then 1 on your touchtone phone. Today's conference is being recorded. If you have any objections you may disconnect at this time. And now I would like to turn the meeting over to Ms. Irene Fraser. Thank you. And you may begin.

Irene Fraser: Hello everybody. This is Irene Fraser. I'd like to welcome you all here and looking forward to our discussion. I'm going to start off here and just give a little bit of an introduction and then Bill Encinosa is going to be talking through the rest of it. And I know that Bill sent out the slides to everybody so I will just be walking through the first several slides and then handing it off to Bill.

So we are here to talk about the funding opportunity announcement on Comparative Health System Performance in Accelerating PCOR Dissemination. So I'm going to give you a little bit of the background and review of the Request for Application. And then as I mentioned turn it over to Bill who's going to walk you through in more detail including walking you through some of the frequently asked questions.

So just by way of a bit of background, the area of evidence-based healthcare is certainly something that AHRQ has been supporting for many years. And that has an important connection with PCOR. PCOR is comparative clinical effectiveness research that looks at the effect on health outcomes of two alternative approaches, and PCOR produces not only clinical findings but also evidence about the effectiveness of different kinds of systems.

So the funding for these projects is going to be coming from the PCOR Trust Fund. And we have specific approval through the ACA, specific authority to support research in this area. In particular, that focuses on the dissemination of research findings for physicians, providers and others to implement PCOR. So the work in this area through this FOA is going to be really essential to that work of making sure that PCOR findings are disseminated and used by those in the best position to do so and in the best possible way.

As some of you may know there was also a recent funding opportunity announcement related to putting findings into primary care practice. This is somewhat of a parallel effort in that it focuses on work through systems, through healthcare systems. So by way of background to that, increasingly physicians are working in more and more complex systems that is not an individual physician or other care provider, but large, very rapidly evolving systems with more complex and diverse ownership structures and other virtual arrangements. And that evolution is occurring more rapidly than our accumulation of evidence on what those new arrangements look like and how they function and in particular how well they might function in accomplishing the outcomes of interest. Therefore, when one attempts to target PCOR and other comparative effectiveness research, to target those dissemination efforts it's less clear than we might like how exactly to do so.

So the purpose of this FOA is to attempt to close this gap. And we will be funding Cooperative Agreements up to three centers of excellence. And each center is going to carry out two different related sets of activities. One is creating a data core that will identify and classify delivery systems and include data on their characteristics and their performance. And then, second, to conduct up to six major projects over the next five years using the data from the data core to compare the performance of different kinds of healthcare systems.

So, in those projects and their relationship to the data core we hope to identify and characterize the systems themselves, what they look like, look at the extent to which they use the evidence. Look at the relationship between that evidence and the take up of that evidence and the production of high quality care and reduced costs, and then to really understand how the system characteristics produce these sorts of outcomes. So, there are two critical terms that we want to identify before going further, one is what we mean by systems and what kinds of systems will be eligible for assistance under this initiative.

And I'm going to turn it over now to Bill Encinosa who will fill us in on the Eligible Systems Study and the eligible kinds of comparisons.

Bill Encinosa: Okay. So what exactly is in scope?

We hesitated to give a definition of systems since there's a lot of new systems emerging that we really don't know about and that's the purpose of this announcement; but, some important guidelines. Eligible systems should include physicians and hospitals, which could involve either informal connections--virtual systems--or formal contracts between these systems. We don't want projects to only look at nursing homes, but if the post-acute facilities are also involved with the hospitals and physicians that would be acceptable.

The key is to always have physicians in your projects on systems. A system based on contracts might include for example physician and hospital organizations, accountable care organizations; or, you might also want to look at virtual systems, systems that really aren't named but where there's definite referral patterns between hospitals and doctors and medical groups. A medical group or IPA that does not formally contract with a hospital but that cooperates with a hospital via informal agreements on ways to improve patient healthcare--that would be in scope.

So these examples are illustrative, not inclusive. But it gives you a sense of what we want to look at.

When you look at a system, you want to look at the types of systems, the incentives within the systems, the integration of PCOR evidence into the system's processes of care, the relationship between the use of PCOR evidence and the quality and cost of healthcare in the system, and then the environment and market in which the system is working--the regulations in the marketplace. We don't accept applications that look at only one of these for example, pay for performance incentives without looking at the broader system or if you're just looking at reducing readmissions that would not be acceptable unless you embed that within the whole system.

System comparisons would involve the whole spectrum of the patient's care. That would be our preference. We don't want very narrowed research areas. We don't want just pediatrics. Only looking at outcomes after joint replacement surgery would not be responsive. But, if for example some systems might have lower overall cost of care for their patients because they provide better cardiac care throughout the whole system then it would be important to look at why that specific example of cardiac care is making a difference.

Irene Fraser: And this is Irene, just to underscore that point. The critical issue is that we don't want to focus on just one narrow type of care, area of care. The joint replacement surgery would be one example and not a whole project just on that. There's  no limitation in terms of ages of the patients or anything like that. But, it's just an issue of a specific type of care. 

Bill Encinosa: In terms of comparing systems we invite all different types of comparisons, comparisons between systems over time, comparisons within one system of what is happening over time within the system;  we welcome all different types of comparisons.

Next, we move onto specific guidance for the applicants. To help guide the projects we'll have a technical expert panel of outside experts, maybe 15 to 20 experts. They will engage with you at least four times a year either here in person at AHRQ or via conference calls. In response to the TEP Meetings the PI will respond and give feedback and maybe adjust some of the methods. One major initiative is for the grantees to work together with the TEP to help harmonize measures for comparing the systems.

We don't really know the perfect metrics for comparing systems so we would expect in the beginning year for the TEP and grantees to work to develop appropriate metrics for comparing systems. We hope the grantees will have early findings to give to the TEP in year one. The first year should be spent developing data for all the projects as well as developing early results on classifying and tracking systems and their characteristics. In addition, to help the grantees we will fund a coordinating center under a separate contract. This is to help synthesize your results and disseminate the results. We expect the grantees to work with a coordinating center to help build a classification system of all these new systems that are emerging over the next five years. We also hope to develop a data compendium of various healthcare systems across the country.

AHRQ will contribute to building that data system, that compendium. We hope to have a broad list of systems across the country, their connections with other providers and their characteristics. And hope to develop a comparison of systems across the nation. For the purpose of the compendia we hope to list systems by name and include identifiers, Tax ID Numbers, NPIs and other ID such as that.

The coordinating center will help develop the data for this compendium and help work with the grantees. Also the center will work with the grantees to synthesize your results and develop policy briefs and data briefs, one to five page briefs that we can use to reach policy people and also systems, healthcare people in systems.

Some basics about the initiative. We're using the U19 mechanism. We expect to have three centers of excellence. The grants are limited to $3.5 million in total cost per year. The project will be for five years. The eligible organizations are as with most of our funding announcements, for profits are not eligible but they can come in as subs.

Woman: (Unintelligible).

Francis Chesley: They are eligible.

Bill Encinosa: A correction, for profits-are eligible in U19s. 

Next, the project—each project can only have one PI. One major requirement is the PI will have to have at least 15% time on the project every year. Also the lead of the data core and the leaders of the individual projects must have at least 15% effort per year.

The page limitations: the data core limitations, 12 pages. Each project you can use five pages per project.

We are looking for a project timeline that will identify major milestones particularly with the first 18 months in terms of getting early results. The review process, first we'll look at criteria in terms of the significance of the project. One main point is whether the project addresses an important problem or barrier to progress in understanding and classifying new types of emerging systems.

Another point is whether the project leads to a better understanding of emerging healthcare delivery systems in order to better target PCOR dissemination. So, a key with all this is whether we can really improve PCOR dissemination within systems.

A second major review criterion is innovation. Does it seek to improve our understanding of how to compare these systems? A lot of systems claim to be very good but we really want to know how to compare these systems in a credible way where we use evidence-based medicine and very advanced methods to compare systems.

The projects should assess the quality and cost of the healthcare system. We want to know metrics of how to identify high performance in these systems.

We also are looking for innovative methods for disseminating PCOR evidence within healthcare systems or looking at projects that address an innovative hypothesis or a barrier to progress in PCOR dissemination. We also care a lot about developing new methods such as dealing with selection issues, how doctors are selected into these systems; the endogenous formation of these systems.

The third main criterion is the approach to the methods. We need a plan project timeline with a big emphasis on early progress. We look for an appropriate budget for the data core activities.

Does the application include a plan to share results particularly Tax ID Numbers and other identifiers to help compare systems?

Does the application include a well-developed plan for requiring data?

Does the data core include statistical methods to compare systems?

We also encourage primary data collection.

Important, how does the applicant plan to assess the impact of systems on PCOR findings?

Do the projects have an approach to look at major impacts on healthcare in the U.S.?

The applicant needs to define the geographical area to be studied and give a justification. Some projects might want to look across the nation. Others might want to look more locally and drill down deeper within a system.

Do projects compare the outcomes of different types of systems, processes, incentives and environments including the market?

Important dates: we expect letters of intent, September 5th. Applications are due October 17th. We expect to review the applications in January. And we expect a start date in April.

If you need additional help you can use the contacts listed in the FOA. For scientific questions you can reach me, Bill Encinosa; for grant questions Dr. Vo; and financial matters, Anna Caponiti.

Next before we open the line we'll go through some questions that we've already received that come up a lot. The first question: does each individual project—remember you can have one to six individual projects--do each of them need to examine PCOR dissemination?

The answer we give is: the goal of the combination of interrelated projects is to address the following question. What kind of systems using what types of processes with what types of incentives and in what kind of markets and environments speed up the diffusion of PCOR findings into practice to improve patient-centered outcomes? So, the applicants should identify how each individual project contributes to this goal.

Second question, does the PI need to lead an individual project? No. The PI just needs to have at least 15% time on the overall grant.

Okay, but an important question is if the PI also leads the data core or an individual project, does the PI need 30%? The answer is yes.

Another question, do you need to develop a common theme for all your individual projects?

We didn't require this because we thought the FOA already has a very targeted theme. The theme is: what are the characteristics of systems that perform well specifically with respect to PCOR dissemination? The projects need to be interrelated with respect to this theme. And the application needs to be clear about that interrelationship.

Another question about structure: a lot of people wonder do we really want systems with a lot of breadth looking at the whole nation over time or do we prefer a more focused structure where people drill down into what is happening within the system? Both are important. We expect both—we entertain both types of proposals. The applicant should indicate the choice that they make and how their application can best contribute to the overall goals of the announcement.

Another question: are you allowed to examine interventions? Yes, if the intervention is already funded. We don't want big interventions proposed to be funded in this announcement.

The final question that we have, page limitations, we had 12 pages for the data core, 5 pages for each individual project. But for the overall research strategy we encourage you to put it in the Research Plan section in the overall component. With this overall component,   there's no page limitation in the U19. But, the PHS 398 encourages researchers to be succinct as possible in their research plan description.

Okay, and now we can open it up to questions that the audience might have.

Irene Fraser: And again if there isn't enough time to answer all of the questions that we answer the morning afterwards.

Coordinator: At this time if you would like to ask a question on the phone line please press star then 1on your touchtone phone. Please unmute your phone and record your first and last name clearly when prompted. To withdraw your request please press star then 2. Once again to ask a question, please press star then 1.

The first question comes from (Steve Shortell). Your line is now open.

(Steve Shortell): Yes. Just a question of clarification, in terms of the description in the solicitation about work to develop taxonomy and so on, is that to be considered, I assume separate from one of the up to six projects, the six projects might well draw on taxonomy work, etcetera, but that in itself would not constitute a project.

Could you clarify that?

Bill Encinosa: We were hoping that would be in the data core. But if you see the taxonomy having a lot of components, I could imagine it might be an individual project.

But my preference, I would expect to see it in the data core.

Irene Fraser: Yes, I agree.

(Steve Shortell): Yes, thank you. That's helpful. Thank you.

Coordinator: The next question comes from (Cheryl Damberg). Your line is now open.

(Cheryl Damberg): Thank you. I was wondering if you could say a bit more about the compendium of health systems across the country. You indicated that the projects would be compiling data including identifiers for the providers in these health systems.

And one of the concerns that I know I've encountered in the past in dealing with health systems and trying to illicit the type of information is much of this information will be considered sensitive.

And I was wondering if AHRQ had thought about data sensitivity issues.

Irene Fraser: This is Irene. The short answer to that would be that we would expect that any Data Use Agreements that are in place would need to be respected.

Coordinator: The next question comes from (John McConnell). Your line is now open.

(John McConnell): This is probably a follow-up question to (Cheryl)'s question. So one of the review criteria is does the application include a plan to share Tax Identification Numbers and physician NPIs.

And so I guess what I'm wondering is so there are nice data sets that are probably possible to acquire to get a DUA under certain restrictions where the analyses would not identify hospital or physician groups per se.

And I can imagine getting those in the timeline to submit an application. Getting something where those data and Tax Identification Numbers and NPIs would be passed onto AHRQ is another level of data sharing that is probably a larger conversation.

So if that is not—so I guess if you have data that you can obtain where it's not entirely clear that the Data Use Agreement allows for passing on Tax Identification Numbers and NPIs to AHRQ, does that mean that that's not responsive?

Francis Chesley: This is Francis Chesley. That would only be responsive if as a condition of award those data elements would be made available through the grantee to AHRQ and so specifically the application would have to include a plan for doing such.

Coordinator: The next question comes from  Hal Luft. Your line is now open.

(Hal Luft): Thank you. And actually two parts to try to clarify within the previous two, one is let's say we could get the Tax ID of all the participating organizations and they're comfortable sharing that but not—so we know who is in the pool but not to have those Tax IDs shared with specific results for specific observations.

So that's the first question. And then the second one has to do with we anticipate that over the five year period new kinds of PCOR findings will be coming out, new ways of applying them will be happening and the members of our team who would be most involved in the analysis of that data may change over time.

So are you open to us identifying who the members are of the team probably in years four and five. The percent efforts will be determined as we get closer to that time period?

Francis Chelsey:    I don't think that that will be acceptable. (Hal) this is Francis again. I think you're going to have to lay out a bit more detail upfront in the application because it will have to have an objective review based on what you include in the application.

(Hal Luft): No. I understand that part. So we list the team members. But we're saying their percent effort may vary and under a Cooperative Agreement AHRQ would be able to review and approve.

Francis Chelsey: Well what I'm suggesting (Hal) and we would have to obviously get in a bit more detail once we see your application, but in order for us to understand for budgeting purposes how your application is going to be put together I'm thinking on this question you need to provide a bit more detail. 

But I would also suggest (Hal) that you submit the question in writing to Bill and then we can answer that and post it on the Web site as well.

(Hal Luft): Right, okay. And then can you go back to the Tax ID Number question?

Francis Chelsey: Say that one again (Hal).

(Hal Luft): So let's say any of us had a set of participating organizations that they'll provide data and we can identify who they are but they are uncomfortable with us identifying—attaching a name to a set of characteristics about that organization.

So we would characterize the sample but we would not be able to tell AHRQ that Plan Number 4 looks like X.

Francis Chelsey: So (Hal) we're going to have to get back to you in writing on this question. It'll be posted on our Web site.

(Hal Luft): Okay.

Coordinator: And the next question comes from (Cheryl Damberg). Your line is now open.

(Cheryl Damberg): Thank you. I had a follow-up question. So you had noted that the projects needed to have physicians in the project. Does this apply for all six of the proposed projects so each individual project would have to have physicians as part of the study or analysis conducted?

Francis Chelsey: So the FOA does not include a statement that all proposed projects, all proposed interrelated projects have to include physicians.

So I think the answer to your question is no. However I think the answer to that question really depends specifically on the individual projects. An application would not be responsive to the FOA if it included individual interrelated project that did not include physicians.

Coordinator: And we have no additional questions on the phone line at this time.

Irene Fraser: Okay. Thank you very much.

Bill Encinosa: We expect to have transcripts. And then also posted answers for the questions where we were not able to give an answer, we'll develop answers and post those on the AHRQ Web site.

Irene Fraser: Thank you all.

Coordinator: This now concludes today's conference. You may disconnect at this time.

Page last reviewed September 2014
Internet Citation: Comparative Health System Performance in Accelerating PCOR Dissemination (U19): Technical Assistance Conference Call Transcript. September 2014. Agency for Healthcare Research and Quality, Rockville, MD.


The information on this page is archived and provided for reference purposes only.


AHRQ Advancing Excellence in Health Care