Skip Navigation Archive: U.S. Department of Health and Human Services www.hhs.gov
Archive: Agency for Healthcare Research Quality www.ahrq.gov
Archive print banner

Improving Quality Through Clinician Use of Health IT

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: https://info.ahrq.gov. Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to www.ahrq.gov for current information.

Ambulatory Safety and Quality

Technical Assistance Call


This document summarizes the technical assistance call for the Ambulatory Safety and Quality: Improving Quality Through Clinician Use of Health IT funding opportunity announcement (FOA). Select to access the FOA.

The technical assistance teleconference was held at the Agency for Healthcare Research and Quality (AHRQ) conference center in Rockville, MD, on January 4, 2007. If after reading this document you have any questions or comments, contact ImprovingQuality@AHRQ.hhs.gov.


Introductions

Good afternoon, everyone, and welcome to our Third Technical Assistance Call. Today, we are talking about the Ambulatory Safety and Quality program and Improving Quality through Clinician Use of Health IT Funding Opportunity Announcements (FOAs).

My name is David Meyers. I'm a medical officer in the Center for Primary Care, Prevention, and Clinical Partnerships here at the Agency for Healthcare Research and Quality. Helping me with this call today are several of our staff members:

  • Joan Metcalfe, Grants Management.
  • Rebecca Roper, Scientific Review Administrator.
  • Yvette Davis, Scientific Review Administrator.

Between the four of us, we bring the content expertise, the budgeting and financial expertise, and the review process expertise that we hope will be able to help you through the rest of this process.

Slide Presentation

We'll be following a set of slides today that was provided electronically before the call began: (PowerPoint® file, 185 KB; Text Version). Slide presentations can be accessed using a free PowerPoint® Viewer. Exit Disclaimer

If you didn't receive a set, don't worry.  We will also be posting the slides as well as the answers to any questions we'll review today on our Web site after the call.

At the beginning of the call, it's important for me to remind everyone that while this technical assistance call is here to assist you, really what we're doing today is reviewing and highlighting the FOA and answering any questions that you may have had after reading the FOA.

If at any time something I say conflicts with what's in the FOA, the FOA wins. It is the true arbiter and everything we say should be a reflection of what's in it.

With that, we can turn to the second slide on the conference call agenda. I'm going to talk about the overall program that this is part of, and some specifics about this FOA on improving quality through clinician use of Health Information Technology (Health IT).

We've prepared some frequently asked questions in the slide set that we'll go over with you. And at the end of the call, we should have plenty of time to answer questions from the dial-in audience. When we get to that point, we'll ask everyone to queue up and ask your questions, then the staff as a whole will answer them as a whole.  We won't go (question-answer-question-answer).

Moving on to the next slide, eligible institutions. This slide is clearly out of order. But it's here so that if you're not eligible we can let you off the call now. Eligible institutions under this FOA include organizations that are public or not-for-profit private institutions. These include healthcare organizations, universities, faith-based and community organizations. Additionally, eligible units of local and State government may apply, as well as Indian and Native American tribal governments or tribally-designated organizations. For-profit organizations are not eligible to lead the application. They are, however, encouraged to participate in consortia or as subcontractors with eligible lead organizations. Additionally, for-profit organizations that produce Health IT systems are encouraged to apply in partnership with a nonprofit lead organization.

Because the focus of this FOA is on improving America's healthcare system, foreign institutions are not eligible to apply as the lead organization. They too, however, may come in as members of consortia or subcontractors.

Moving on. This initiative that we're speaking about at this hour is part of a larger ambulatory safety and quality program here at the Agency for Healthcare Research and Quality. The overall purpose of the initiative is to improve the safety and quality of ambulatory healthcare in the United States.

We recognize that much of our earlier work in improving patient safety and quality in the United States was focused on the inpatient setting. And at this time, we're not—we are continuing that work while, at the same time, through this program, expanding it to the increasingly complex ambulatory care setting.

As you'll note, a large part of the ambulatory safety and quality program has an interest in harnessing the power of health information technology. This is bringing together two of our large portfolios of work in quality and patient safety, and Health IT.

If you'll click to the next slide, you'll see that this initiative has four components. And, this afternoon, we're talking about Improving Quality through Clinician Use of Health IT. For those of you who have been with us earlier in the day, I do apologize if some of this is repetitive, but we do have some new folks on the call. 

In these FOAs, we define ambulatory care services as the types of health services provided by healthcare professionals on an outpatient basis. For these FOAs, we note that this setting may include the healthcare clinician offices, both large and small practices, outpatient clinics, community health centers, emergency departments, urgent care centers, and ambulatory surgery centers. While home care is often not typically included as part of ambulatory care, for the purposes of this FOA, home care entities are considered ambulatory and are welcome to apply.

RFA-HS-07-006 has quite a—you'll see quite a large number of aims and goals. And that's because, as we'll say later, we recognize that this spans the spectrum from discovery to translation to measurement and then to widespread dissemination of strategies. We're interested in projects that investigate novel methods and projects that evaluate existing strategy for clinician use of Health IT in the ambulatory care setting.

Earlier today, we had a call about patient centered care. That FOA has the patient as the focus. This initiative puts the clinician as the focus of its work. The whole health care system also is part of this initiative.

We're looking to improve outcomes through more effective clinical decision support. So clinical decision support, medication management and care delivery are all focuses of this FOA. Additionally, we are interested in projects that examine how Health IT fits into workflow and how the effective use of clinical alerts and reminders can be integrated into health care system delivery.

Moving on to the next slide, more specifically on the themes where AHRQ in this FOA is focused. These include the impact of Health IT on outcomes in the ambulatory setting, and across high-risk transitions of care, looking at novel approaches to providing high-quality, appropriate care through the use of Health IT, and the essential strategies for safe, successful and productive Health IT adoption in the ambulatory setting.

On the next slide, entitled "Areas of Interest," we note that the core theme for this FOA is the clinician's use of Health IT to improve quality. And we have some illustrative questions including providing information and decision support to clinicians, clinical workflow and cross-disciplinary teams (that comes up several times in the FOA and is clearly an interest of ours) as well as care coordination.

The next slide brings up the idea that this application is done on the SF 424 (R&R) form. We'll talk more about the specifics towards the end of the call. But like in most grants—research grant applications—we expect that:

  • You'll describe the specific Health IT application and intervention that you will study.
  • You'll develop the project design and the methodology plan, which includes research design, specific outcome measures and an evaluation plan.
  • You're required also to have your milestones and there is a requirement to include a timeline.
  • You'll assess the primary and secondary outcomes through a robust analysis plan.
  • And then we expect that there'll be a plan for dissemination included in your project.

The next several slides just highlight—again, all of these come directly out of the FOA—some of the requirements of the research design and methods section. And I'm just going to point out a few specific things on each of these slides. So these are the highlights of the highlight.

On the first slide, the applicants should describe the development of the intervention and the methodology for implementation. Specifically, when possible, the Health IT intervention should conform to interoperability standards and use certified products. And if folks have questions about that, we can talk more at the end.

On the next slide, you'll see that in addition to specifying the primary and secondary outcome measures that you would like to use for your project, in the areas that your project will move forward, AHRQ is requiring, when appropriate, some additional measurements. Please note, these are not supposed to be the focus or necessarily will be the focus of your work. This is what AHRQ can use to look across all of the projects it runs in this FOA as part of the larger program, as part of our Health IT and patient safety projects as an agency, and as part of the Department of Health and Human Services.

So, we are requiring, when appropriate, that you give us some basic measurements of the adoption of Health IT. They're specified here. There are five of them. They include:

  • The percent of adoption and use of Health IT in the settings with which you work.
  • The percent of patients who receive appropriate care for prevention, treatment and medication therapy.
  • The percent of eligible patients within practices who have access to their personal health information.
  • Patients' and providers' access to, and utilization of, quality measurement reports and especially if those are aggregated through an Health IT health information exchange.
  • The percent of ambulatory clinicians within the practices with which you are working that routinely use measurement tools to evaluate their patients' experience with care.

We give you one way of doing that with our CAHPS® clinician and group survey tool. Again, we are asking you, when appropriate, to tell us how you can report to us data on these measurements.

On the next slide, there are two things I'd like to point out. One is that we are interested in you and your application talking about the future sustainability of your intervention after the grant program or the grant funding has ended. And finally, all projects are required to include a dissemination plan. The dissemination plan needs to be both in the narrative text of your document and incorporated into your proposed budget.

We're very happy to announce that we believe there'll be many successful projects that we would like to have additional dissemination, above and beyond what has already been proposed, and that we would like to have a limited competition for additional dissemination grants for folks under this application. However, your application to get into the program has to be done without using those funds for your budget. So your dissemination plan needs to stand alone and make sense without this additional funding. But just to think in the future, we're going to help disseminate—we're going to ask you to help us disseminate further the results of your work.

In the next slide, I'd like to highlight our concern and the importance that we place on privacy and security. All applicants are required to describe how, both in the development and the implementation of their project, privacy and security issues related to the exchange of sensitive health information will be identified and addressed. The FOA gives some very nice descriptions of what we mean by this, some sample ideas of how this can be done, as well as links to resources to further think about the issues in privacy and security. And as you will see later, how you've addressed privacy and security issues is one of the criteria on which all applications will be reviewed by the peer review committee.

On the next slide, we highlight that this FOA has three special funding preferences. These are not priority areas that the review committee, the committee of your peers, will be asked to speak to. These are going to be criteria that AHRQ uses in determining how we use the pools of money that we have set aside to fund this FOA.

There are three of them, and I'll start by noting that we expect many projects will come in under more than one, if not all three, of these funding preference areas. To do that, you'll follow the instructions in the FOA—these are detailed in Section 3.3—and that you will include this information not in the text of your application but in Appendix 1. Even if you choose not to come in for one, two or three of them—if you decide you're not interested in being considered for any of the funding preferences, we ask that you also have an Appendix 1 and that you include the sentence, "No funding preference requested." If folks have more specific questions about this, we'd be happy to talk about that at the end.

The next slide points out to you that AHRQ is going to use two large considerations to make its decisions about the funding of applications under this FOA. The first is the scientific merit of the proposed project as determined by peer review. This is the area that you have control over. This is what you say in your application and it's material that will be reviewed by a committee of your peers. And we'll talk on the next slide about what specifically they have been tasked to look for.

But the rest of the slide points out to you that there are many other areas that AHRQ will consider when making its final decisions about funding considerations, including how your projects fit into AHRQ's larger goals, other projects being funded within the Agency and the healthcare needs of the country as a whole.

On the next page are the specific review criteria that are outlined in the FOA. These are the headings. I strongly encourage you to read the specific questions that this review committee will be given and charged with when reviewing your application. They're categorized into six main areas: significance and impact; usefulness and generalizability; approach, which includes methods; the investigators and the partnerships brought to this application; the settings in which the study is proposed to be conducted; and your budget. I point out that many people have not seen the terms "usefulness" and "generalizability" used as review criteria in the past. So, take a look at those.

Additionally, there are other considerations which the review committee will be asked to discuss. Those include your degree of responsiveness to the FOA as a whole in three specific areas worth noting—privacy and security (which we already mentioned), protection of human subjects from research risk, and inclusion of priority population. It is to your advantage to make it easy for your reviewers to understand how you are addressing these three specific areas in your application.

Additionally, when you complete your SF 424 (R&R), there are other requirements that all applications to the Department of Health and Human Services are required to fill out—additional language on human subjects protection as well as the inclusion of women and minorities. There are specific tables that you're asked to complete. Your review committee, while not using it in your score, are required to state whether or not you've successfully addressed those. If you do not, you cannot get funding. So please, pay attention to all of these things.

The review experts in the room wanted me to correct something I said. Those additional things in the SF 424 can in fact be used by the reviewers when they consider your application because it's under your "additional considerations," and can be used to downgrade your score if you don't address them satisfactorily, as well as grounds for not funding the application outright.

On the next slide, you'll see our mechanisms, some details about the FOA. These are just the basic things. This is an R18 mechanism. It's a research demonstration and dissemination project mechanism, the goal of which is to provide support designed to develop, test and evaluate health service activities. This is a one-time solicitation. At this time, AHRQ has no plans to reissue this. You may want to take that into your planning.

We projected $9 million in total costs in FY 2007 under this FOA which would fund between 20 and 40 grants. All applications must not exceed 36 months in duration. The total budget cost over the period must not exceed $1.2 million, and the budget in any one year may not exceed $500,000. Failure to meet any one of those three criteria are cause for the application not to be reviewed (and to be returned).

A few more details.

On the next slide, we say that AHRQ does not accept modular budgets. We only use the detailed research and related budget, which is what you'll find in the SF 424. We do not use the PHS 398 modular budget, and any application that attempts to use that or submit that will be returned without review.

Specifically, the SF 424 has replaced the PHS 398, and it's an electronic application. All applications under this FOA must be submitted electronically using the forms available and for this specific FOA. There are details in the FOA itself about how to get the electronic forms as well as how to register your institution and the principal investigator (PI). And that's worth a little extra time for those who have not done this before.

Registering is required before you submit your application, and the process of registering an institution as well as the PI can take up to six weeks to complete. The institution must register in two places and the PI also must be registered in one. Details about that can be found on the AHRQ Web site under the section under Funding Opportunities and Information on Electronic Application. Additionally, there are links in the FOA itself.

Going on, a few last slides. We've already talked about who's eligible. On the next slide, we talk about the principal investigator. The details of the skills and qualifications of the PI are detailed in the FOA and highlighted here. I'd like to point out that the FOA says that the PI should devote a considerable portion of his or her time to the project. If they will devote less than 20 percent time, the application must then include an explicit justification.

Whenever the FOA says "must" that is a review criteria which are our grants team will look for. They will go through all applications and make sure that all of them must have been done. Applications that do not comply with this may not advance to the peer review process. So it's worth taking a look at what the musts are. And most of those have been highlighted in today's presentation.

On the next page, you'll see the key dates for this FOA. Letters of Intent (LOIs) are due by January 19, 2007. Details about how to submit a letter of intent and what we are expecting are in the FOA. I'd like to point out that this is a really important service that you can do both for us and for yourselves. Submitting a letter of intent is not binding, but it helps AHRQ understand how many applications we may receive, what types of work we should be ready to evaluate, and those two things help us plan who should be on the special emphasis panel that will be reviewing your grant.

So, if you let us know what you are up to, we can make sure there's somebody who knows how to speak your language. If you don't let us know, we may be surprised and that might decrease the quality of the review you will receive. So please, consider submitting an LOI. You can fax it to us, you can E-mail it to us, you can send it by mail, but send us a letter of intent.

The application submission date, the date these are due, is February 14, 2007. You will note in the FOA itself it also has something called the closing date. The closing date is just when the computer system shuts down. It is irrelevant for your purposes. What you need to know is that your application is due on February 14, 2007.

We anticipate peer reviews will take place in late spring of 2007. There are many factors that may change that. Using that as our guide, we'd like to tell you that your earliest anticipated start date when you put together your budget and timeline should be July 2007.

With that, we're going through a few questions that folks have asked us in advance before we turn it over to the group.

So flipping on to the next slide, "Are hospitals allowed to apply?" I'm sure most of you know the answer. Yes, public and nonprofit healthcare institution, including hospitals, are eligible to apply. For-profit organizations, including for-profit healthcare organizations and Health IT vendors can be included in consortia or as subcontractors but cannot lead the application.

On the next slide, "Can an application be submitted for all four ASQ funding opportunities?" And the answer is: an institution can submit applications to more than one Requests for Application (RFA), and each of these RFAs will be reviewed by a different study section. That said, you may not turn in the same application to more than one FOA. Suppose you're working on clinician-patient communications. So it's both patient-centered and clinician data. You've got to make the call about what outcomes are you most interested in and applying only for that FOA. But from an institutional perspective, if different teams are working on different projects, they may apply for more than one application as an institution.

On the next slide, "Does AHRQ accept modular budgets?" I hope the answer is clear, but it's worth repeating. No. Applications received with a modular budget will be returned without review.

On the next slide, "Are emergency rooms or emergency departments ambulatory care sites?" For the purpose of this FOA, absolutely, please join us. We list again, as we've described earlier, the different settings that are involved, and note home care agencies—and care delivered in the home are included under this FOA.

On the next slide, a very often asked question is, "Does AHRQ accept co-PIs?" The answer is no.  AHRQ requires that the lead institution designate one and only one individual as the project's principal investigator. If you'd like to give other titles to other members of the team such as co-investigators, that's acceptable. But there needs to be one primary principal investigator that's listed in the application.

And the next question, "How much detail should the letter of intent include?" Again, we use this to estimate how many peer reviewers and what expertise we need. We do not give you any feedback on the letter itself. It is nonbinding. The content that we request from you is described here as well as in the FOA. It is not required if you don't get it together in time. That's okay. And I'd also say, if it's the 20th of January and you realized you haven't sent it, send it anyway—it still helps us. But we're on a time deadline here, so we request that they come in here as early as possible.

Question and Answer Period

All of today's discussions and the answers as well as the slides will be posted on our Web site, and we will try to send an electronic notification to everyone who's registered when those are available for review. If your question doesn't get answered today, you can submit your questions to us electronically at ImprovingQuality@AHRQ.hhs.gov. That E-mail is on the last slide as well as in the FOA. Any questions received by the end of today, we will try to include in the transcripts as well as have answers for them.

Finally, we've discovered, this being our third call, some questions and comments from one FOA are relevant to another. So it may be beneficial for you, even if you are only interested in this FOA, to review the technical assistant questions of the other FOAs to make sure you've got the whole picture.

Responses to Presubmitted Questions

Role of the Principal Investigator

Question: What types of education or experience are expected for the PI?

Answer:  We refer you back to the FOA, where there is a section on eligibility of PIs where we described what is expected. There are many different people who can fill this role. It is expected that the PI will not have expertise in all aspects of any particular project but that the PI will coordinate the work of the team. And over the course of your application, you'll explain how the team works together to meet and be able to do all of the things that you're setting out to do.

One requirement is that the PI must or is encouraged to devote a significant portion of his/her time, defined in this FOA as at least 20 percent. If you believe your application is strong even without your PIs devoting 20 percent of their time, you must include a detailed justification for why you believe that this project can still go forward in that type of structure.

Question:  Can we have the same PI on two projects and can that go forward?

Answer:  Well, you've heard the requirement that the PI be able to devote 20 percent of his or her time to the project. If the PI has the time to give 20 percent to Project A and 25 percent to Project B, that's acceptable. If that would push them over to the Federal limits and they're now working 122 percent, that's not acceptable. But the same PI can be listed on more than one application. You should note whether they have the time, if they were so fortunate as to be funded under both applications.

Question: How do you select the PI?

Answer: There is a combination of researchers, clinicians, and vendors and who's the right person? There's no specific answer. It's your job in the application to justify who you've picked and how they will coordinate the team and how the efforts of all the members of the team come together to give us the complete package. So it can be an M.D., a researcher, or none of the above; it can be the person from the ambulatory setting, the research setting, or even the vendor setting, if all the other requirements of the grant are met.

Question: The application notes that the PI must or should be affiliated with the applicant organization. Does that mean that the PI must be employed by the lead applicant organization? 

Answer: From our perspective, that is the easiest application to process and is most easily acceptable.

If the PI is not employed by the lead organization, the application still may be considered. It is critical, however, that the application explains how the lead organization, who is entering into a fiscal responsibility relationship with AHRQ, can guarantee accountability of the actions of this person. So, showing a defined, contractual relationship is one way that that might be achieved. And good documentation and description of how decisions are made and people are accountable and that the projects will move forward is going to be required and looked at very closely in an application where the PI is not employed by the lead institution.

Vendor-Based Systems

Question: Is there any issue to consider in regard to implementing a technology on a vendor-based system? 

Answer: While clearly, there are multiple issues, I think the main question here was, is it okay to move forward or to have something that is a commercial product that's already in existence? And the answer to that is yes and there's even language in the FOA that supports that and encourages people to work with things that have already been done and explore furthering their use. The use of a vendor-based or proprietary software, hardware, or system is acceptable. However, the vendor in that case, if they are a for-profit institution, cannot be the lead. But under this proposed scenario, it would probably be a subcontractor and therefore okay.

Methodology

Question: Is a particular methodology such as the randomized control trial preferred for the application? 

Answer: No. The FOA is an R18 and a research application. But specifically, we ask that your methods are the best methods to achieve the outcome that you've set out for yourself. These aren't simply implementation grants, it's not just "can-you-make-a-system;" you do need specific outcomes that will be measured in an analysis plan and there must be a research structure. That said, however, the FOA gives you clear guidance to pick whatever methodology makes more sense for the project that you're working on, and then give that justification to your peer review committee.

Budget Concerns

Question: Are there any potential restrictions placed on the budget? More specifically, Can the money be used for infrastructure or is it to be used for evaluation only?

Answer: In this grant, funding from the AHRQ portion of the budget may be used for hardware, software, research, consultation services, or even for things involved in the implementation of Health IT. However, there is no set figure or percentage outlined in the FOA. With that said, I recommend that you look at the review criteria and note that your application will be reviewed on multiple criteria including the budget. And, specifically, your peers will be asked to review whether the budget you've proposed is reasonable and appropriate for the work that you've set out to do. And they're also going to be saying, are the outcomes that you are proposing and the methods you have for evaluating appropriate?

So the grants that are very high in implementation with no evaluation, while they would meet the criteria, may not be favorably reviewed based on their outcome. A project that spent a lot of money on technology may not be considered in the budget section as being responsive to the FOA. However, that budget is for you to decide and to make the best case you can for how you spend your money. The review people also have asked me to say that any money spent on technology that is proposed in the budget needs to be justified in the budget section as well as in the description of why it's necessary in your research plan.

Question: Can funds be used retroactively to pay for an Health IT system that was previously purchased?

Answer: That is not acceptable. Funding is only prospective and begins on the date of your start date based on if you get an award. It'll be specified in your award letter. However, that question went on to say, what if we've got a contract that's ongoing? Could we pay our next month's technology bill that's our contractual obligation using this fund? If you are funded and you've given the justification and your peers say yes, that seems like an appropriate use of these funds, that potentially would be acceptable to the extent you prorate it, that is, to the present time, and to the extent used by this project.

Question: Can a State primary care association (PCA) that represents federally qualified health centers apply under the vulnerable populations funding preference on behalf of its members?

Answer: I would separate that out into two parts and say absolutely, any State PCA, if it is a nonprofit institution that meets all the other Federal requirements to receive funding, most certainly can be a lead applicant. They would be able to apply any special preferences based on their work; they would apply based on the settings in which the work is going to be done, and then could list their federally qualified health centers. The qualifications are specifically listed  in Section 3.3, including "are they federally qualified health centers?"  And they could list them and that they are qualified and therefore request funding preference.

Projects Yet To Be Implemented

Question: What if we have our electronic medical records (EMR) system chosen but have not implemented it yet? We don't have an EMR system in place today, but we will in the future, so can we write an application? 

Answer: Yes, absolutely. However, there's a big contingency with that. If you can't give the specifics about how things are going to be done, your reviewers may not be convinced that you have the ability to get it done and measure the outcomes that you've stated you will do. So the more definitive you can be in the application, the more definitive you can be about your methods and your outcome. However, if you're in process or beginning to implement, you can include that as part of this project and in your description. So that's a yes—but.

Open Question Period for Conference Call Participants

Question: Please make the distinction between the Improving Quality FOA and the Patient-Centered Care FOA, because clearly, these two have the potential to overlap.

Answer: The Patient-Centered Care FOA puts as its central focus the patient's experience of care. It's harnessing Health IT to improve quality and safety, but really to improve the patient's experience of care. Therefore, the primary outcomes of such a study are going to be the patient's experience of care, the patient's self-management skills, or definitions of patient-centered care.

This FOA, Improving Quality, puts as its central focus the clinician and the clinician's ability to improve quality by using health information technology. Its outcomes may be patient health outcomes, efficiency outcomes, quality outcomes, or it may be others or the outcomes described in the FOA. The difference is not so much in what the intervention is, the same intervention could be done in two different ways, it's from whose perspective you are measuring. Clearly, somebody could come up with an idea that would wrap around both, and if you do that, we're asking that you pick which one you are going to put as primary. Look at your primary outcomes. If it's the patient-centered measures of care, then you would go under RFA-HS-07-007; if you're looking at improving quality health outcomes, quality outcomes, or efficiency outcomes from the system or the clinician's perspective, that would be under RFA-HS-07-006, the one we're discussing at the moment. If you have additional questions, feel free to contact us about that.

Question: My question relates to the mechanics of putting together the application. Can you clarify where letters of support should be provided in the electronic application? Is it under Item 16 of the research plan, or as one of the attachments in the Appendix (where I have instructions) in both places?

Answer: The attachment section is a good place for most kinds of letters of support. If there is a letter of support that rises to such an important level that you'll need it to be part of your research plan (it's not intuitive to me what it would be but if it was), you would be able to include that as part of your research plan, but the FOA describes how those are the attachments, and what type of attachments, and letters of support in general belong in that section.

Question: Is the inclusion enrollment report under the research plan relevant to this particular funding opportunity?

Answer: A report that's required of all grant applications under the SF 424 in Federal health agencies. It appears in your SF 424 in its correct place. You should complete it; it may be used in the review process. However, it does not add to your page limit for your research section plan. That said, how you handle human subject protection, privacy and security is a larger issue that may in fact need to be part of your research plan and specifically part of the evaluation criteria.

Question: Also, regarding the resource sharing plan under the research plan, is that essentially the same thing as the discussion of a plan for dissemination?

Answer: Page 190 of the SF 424(R&R) Guidelines describes what a resource sharing plan is. The guidelines describe a specific aspect of data sharing for large grants over $500,000 in direct costs, which does not apply to this FOA (where the maximum of $500,000 applies to total costs.) The resource sharing plan is different from a dissemination plan, which addresses the activities of the project in sharing its findings, lessons learned, and/or tools produced.

Question: Are patient-centered outcomes or patient measures necessary for this application?

Answer: You can look specifically in the review criteria to see they are not required unless appropriate. They may be appropriate; they may be the best measures for some projects. Other people's projects may like to look at other outcomes. For example, efficiency outcomes in general could also be considered acceptable, and examples are given in the review criteria that you may look at more.

Question 1: Under the slide, "Mechanism of support and Funds Available," the bullet point that says "The budget also may not exceed $500,000 in total cost in any single project year," that $500,000, is that AHRQ funds or is that our budget?

Question 2: If our total budget exceeds what is available in the grant, how are we supposed to specify what portions of the budget are going to be coming from our organization versus what we're going to use from the grant money?

Answer: Two questions were asked about the total budget and how to apply. They were wonderful questions that get to the idea that AHRQ encourages in-kind support from vendors, from institutions involved in the application. That is looked upon very favorably. In your 424 budget, however, include only the funds that you are requesting from AHRQ. You will then have a budget justification section. The budget justification section is where you can put those kinds of in-kind contributions and note them. That answers our next question of how do we show the $500,000 in 1 year. The budget per year that you're going to submit is only for the AHRQ-funded portion. But again, in-kind support is encouraged and should be included in your budget justification section, and even in your research plan or description of the plan.

Question: There were five measurements, outcome measurements, that were indicated on the slide. On one of them, what if there is no measurement tool in use to evaluate a patient's experience? Can one start with the grant to put that into effect?

Answer: From that general question, the FOA says, "when appropriate." If it is not applicable to the work you are doing and impossible for you to do, state that; that is an acceptable answer.

However, the question specifically asked about Measure 5, which is about the patient's experience with care. AHRQ strongly encourages that you use AHRQ's CAHPS® measurement set for evaluating patient's experience with care. If your clinicians are currently using another instrument and you want to tell about that, because the question says, "our patients," you could answer, Part A, how many of our providers are measuring it at all, and then could ask, Part B, how many are using the AHRQ CAHPS® measurement. That would be a lovely way to handle it.

Question: Will there be other patient measures coming out of this FOA, and Are we providing other ones?

Answer: No. We strongly recommend your using AHRQ CAHPS®.

Question: Could you give us a little more clarification on the dissemination plan as it relates to Health IT? Are we are talking about the findings? Are we talking about that which has been developed? If you can just give us a little more clarification on that, I'd appreciate it.

Answer: Specifically, are we asking you to tell the world about your findings? Yes, we are. That is generally what is considered to be part of the determination plan. It can also be about their broader interpretation; you are welcome to tell people about the methods you used, your experience and lessons learned in the implementation process, and disseminate the other learning(s) that took place. You'll find those would've also been on this part of your findings.

There is not much description of the dissemination plan in the earlier part of the FOA, but if you'd like to understand what AHRQ is interested in, look on Page 23 of the FOA, where we describe and give you the questions that the review panel would be asked in reviewing your application. Under "usefulness and generalizability," they give you the ideas—the specific questions there describe the kinds of things AHRQ wants you to be talking about in your dissemination plan.

Question: Our local Practice-based Research Network (PBRN) is participating as one of several ambulatory sites. Would that be listed in the funding preferences section or do they have to be the lead organization?

Answer: Does the PBRN need to be the lead organization to qualify for the funding preference? No, it does not. The lead organization must be a non-profit et cetera. If a PBRN is participating as a partner they should be mentioned in the funding preference section, but if they are one of a 100 partners, we would need to know that. If they are 60 percent of the sites, we would want to know that, to understand how you meet this funding preference.

Question: I just wanted a little clarification about what budget funds can be spent on, whether they primarily have to be evaluation or whether they could also include supporting the IT intervention?

[See other budget-related responses.]  I think we've answered that.

Question: I have two questions. One is regarding home care—what kind of medicine is covered? Second, how much of the budget can be used for development of the IT portion itself?

Answer: Absolutely.For the purposes of this FOA, care delivered in the home is considered ambulatory care; however, there needs to be an organization that is ambulatory care focused involved in the application.

[See budget-related responses for second part of question.]

Question: Slide 11 suggests that the applicant has to be an ambulatory health provider or be implementing the Health IT system in a provider organization. What would the status be if a nonprofit is implementing but not in a provider organization, but in free-standing offices?

 Answer: There needs to be a nonprofit organization as the lead and overall in the project. You must be working in some way with ambulatory clinicians or ambulatory providers. So, if you have a lead that's a non-ambulatory, but they're working with ambulatory, you're okay; but the lead still has to be the non-profit.

Question: If we're fortunate enough to receive funding from you from two different parts of the Ambulatory Safety and Quality Program, would we be able to use the same PI for both projects? 

Answer: [Go to: Presubmitted Responses.]

Question: Is a project applicable that would focus much of the cost and the activity on developing the IT application, and the evaluation would be through a subcontract? And, if that is acceptable, is the PI the project director who is developing and implementing the IT application, or is it the subcontractor who's actually researching and evaluating the impact of that application?

Answer:  [Go to: Presubmitted Responses.]

Question: Are pharmacists and ambulatory clinician providers using Health IT to improve the quality of care an acceptable group under this FOA?  

Answer: If your project applies to the pharmacist and will show their ambulatory clinicians are using Health IT to improve the quality of care, that's an acceptable group under this FOA.

Question: We work in a collaborative relationship with tribes in a three-state area under the direction of a tribal health directors' organization and an overarching board composed of tribal leaders. And we do work collaboratively with university systems in the area of research. We also do quite a bit in the area of epidemiology and in financial planning. Is it good enough to have a collaborative relationship even though we may not be a provider per se?

Answer: If the lead is a non-profit and you're working with ambulatory clinicians or in an ambulatory setting, you are able to apply, so that grantee should feel free to apply.

Question: Yes. I had two questions. First, could you clarify what is meant by machine-actionable?  And second, could you speak to your preference for an randomized controlled trial (RCT) model versus a demonstration project with an outcome?

[Go to: Presubmitted Responses for second part of question.]

Answer: The FOA uses the phrase "machine actionable information to improve quality." It is a term of art. As you look through this whole field, we believe you will see that some of those definitions talk about the idea that a computer could have the capability to use a piece of information, translate it, and add to its value.

Question: Can grant funding be used retroactively to purchase any equipment or software or, for that matter, any ongoing payments to already contracted Health IT services if these prior purchases or continuing payments relate to the grant?

Answer: [Go to: Presubmitted Responses.]

Question: Health IT adoption by patients or physicians was heavily influenced by behavioral aspects for both. Is there room in these grants to evaluate behavioral aspects? How will reviewers consider that as part of the evaluation plan?

Answer: That would be acceptable. Just talking about how people think and their interactions and their behaviors around computers, without relating it to the ability to improve quality, would not be acceptable. We refer you to the significance and impact review criteria section and the questions asked there as well as the usefulness and generalizability section.

Question: Please speak to the distinction between the primary and the secondary outcome measures you're looking for. And also, how these should be related in the narrative to the five outcomes that are required?

Answer: This is an important distinction to make. The primary outcomes are what you're really interested in, as well as your secondary outcomes. You're going to write a research plan about those outcomes and how you're going to measure them. You'll talk about how your intervention is going to make a difference in the quality and how you therefore are going to measure the differences and show an outcome change. That whole set has to be done and it's a complete package.

We are asking you to report to us how you can report data on the five measures. These are not what your intervention needs to be focused on. Nonetheless, there are things that, as you go along in your business, we are asking you to collect data about and feed them back to us. They do not have to change. You may report zero even though zero is the worst answer to a question or you may say 97 percent. We're not going to grade you or evaluate you on what the number is; but we do need to see in this application that you're willing to help AHRQ collect the standard measures that are being used to compare across multiple initiatives. If you have additional questions about that, please let us know.

Question: When we register our PI, will we be told for certain how long it will take to process? Because right now, we're about 6 weeks out from the deadline, and if it takes 6 weeks to process, we probably shouldn't go ahead with our application.

Answer:  I said that registering may take 6 weeks and that's why this call is occurring now so you actually can start. It rarely takes 6 weeks. It will take 6 weeks if you make lots of mistakes and have lots of complications. It took 6 weeks closer to the beginning of the registration when, many institutions around the country were suddenly trying to register. Because this process is now well under way, the turnaround time can be as quick as 1 week. So, I would encourage somebody who hasn't yet started the process to start it now but if they're interested in applying, not to give up, this can be done.

Question: This is our first opportunity to submit a National Institutes of Health (NIH) research grant and we were wondering if there is any possibility of reviewing past applications—some idea of what it looks like when it's finished?

Answer: Unfortunately, we don't have that. There are some videos available on the AHRQ and NIH Web sites about the application process that are essentially tutorials on applying and preparing a grant application. They are very useful for explaining the different sections, writing a budget, preparing the study section, the type of feedback you can expect. 

Question: Our clinics currently don't have an EMR in place, but our IT team is actually in the final stages of selecting our EMR system. Am I correct in saying that we already need to have this EMR system chosen in order to complete the application? The application can't be filled out with just a general broad base as it needs to have details specific to the exact system and its components? Is that correct?

Answer:  [Go to: Presubmitted Responses.]

Question: Please provide more information about the security and privacy language. 

Answer: AHRQ is certainly not interested in proprietary secrets. Additionally, if you work with some group, and there was something you determined was proprietary, if there was a "freedom of information act" request, you're given an opportunity to address that as your personal trade secret. So, don't worry about that.

If you look at the FOA, and again, I recommend everybody do so, the section immediately preceding Section 2 and above the paragraph on outcome metrics, there are two paragraphs that describe that we're talking about policy. We want to know that systems are put in place to ensure that privacy and security matters are continually addressed and reassessed. Now clinicians are broadening the partners in this process, and even if nothing ever goes wrong, the right policies should be there to ensure that as things change in the future, these issues are being incorporated into your planning and redevelopment. We want to understand how you began to think about incorporating privacy and security when you developed your plan.

This is a policy-level question that we encourage everyone to more actively think through. There are some wonderful resources within the FOA to help you further your thinking, as well as specific examples to explain the kinds of things we expect to see in this section.

Closing

We will pull all of these questions together and provide a transcript as well as copies of the slides to you electronically.

Letters of intent are due on January 19 and the FOA is due in February. 

We look forward to hearing from many of you. Thank you so much. Good night.

Current as of January 2007

 

The information on this page is archived and provided for reference purposes only.

 

AHRQ Advancing Excellence in Health Care