Skip Navigation U.S. Department of Health and Human Services
Agency for Healthcare Research Quality
Archive print banner

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to for current information.

Web Conference: Asthma Return-on-Investment (ROI) Calculator: Transcript (continued)

Margie Shofer: Great, so I'm going to now turn this over to Elizabeth Stranges for the demonstration.

Elizabeth Stranges: O.k., thanks, Margie, and good afternoon. This afternoon I'm going to show you how to use the Calculator to determine return on investment of your quality improvement program. And the questions were great and you'll see a lot of the things you asked questions about covered as I go through.

Before I begin the demonstration, let me mention that when the demonstration is initiated your captioning panel will drop from your view and a toolbar will appear in the lower right corner of your screen. If you wish to reopen your captioning panel, click the icon farthest to the right on the toolbar next to the question mark icon.

I'm going to go into Sharing, and I'm going to walk you through two scenarios with the Calculator today. In the first, I'll explain the basic features of the Calculator and how it synthesizes the data from the meta-analysis that Ginger was telling you about and how it brings in other default data that are resident in the Calculator to produce an estimate. In the second, I'm going to show you how and where to enter your own data into the Calculator to generate an ROI estimate that's specific to your quality improvement program.

Everybody should see on the screen right now the Home page of the Calculator, Asthma Return-on-Investment Calculator at the top of that page. The text on this page just provides a brief written overview of the Calculator. I want you to turn your attention to the orange bar on the left side of the page. This is the step-by-step guide to the Calculator. This is going to be present as you navigate your way through the Calculator.

Section One is the introduction and that provides you with some basics about getting started. Feel free to look at that after the Web conference is over. Also, Section Five has several resources, a user's guide and a technical guide, that you can look at at your convenience. Today we're just going to focus on Sections Two, Three, and Four, which deal with the actual operation of the Calculator.

The first step in using the Calculator to generate your estimate is to define for the Calculator the key features of your quality improvement program. So I'm going to click over here on the Required Information link. You should all see on your screen right now a Required Information page. This is where we're going to enter these key program features. We need to tell the Calculator, first of all, who's eligible for the program, who are we going to target. We're going to tell the Calculator which types of studies should be used to estimate the impact of the asthma program. Finally, we're going to tell the Calculator what costs and savings should be included in the results.

Let's start at the top here where it says who will be eligible for the asthma program. For both scenarios that I'm going to take you through today, we're going to focus on the Alabama Medicaid population. Here you see that the type of insurance is the first thing you need to select and you have three different choices. The Calculator is populated with data for each of these insurance types, and we're going to select just Medicaid. We're going to select Alabama. And we're going to focus on children and adults. You can select one or the other or both. Speaking to the question of asthma severity, we have these three choices and in our first scenario we're going to select all asthma. That's any patient who has a claim with a primary or secondary diagnosis of asthma.

I'm going to select Randomized Control Studies as the study design. You can see that there are drop-down menus here so you don't have to choose that but that's what we're going to go with for today. Also, if you're using the Calculator, anything that is underlined is a link, so if you have questions about any of these things as you're going through, click on it and it will take you to the appropriate section of the user's guide.

The last section down here is what costs and savings should be included in the results. For Type of Costs, you can choose to include just the cost of asthma treatment or you can choose to include all of the medical costs associated with the asthma patients. We're going to go with Asthma Treatment today. Another question that came up is which cost perspective to look at. As Ginger said, we have the option of just focusing on the costs to the program - in this case Medicaid or the health plan - or we can include the patient costs as well, so any copays. I know there was a question about the prescription drugs, so any cost to the patient or that kind of thing would be included here. So we're going to go with this program and patient cost perspective.

Finally, what do we want to include in the final results? Do we just want to include health care savings or do we also want to include productivity gains? You can select whichever of these you want. We're going to go with both for today. Once we've made our decisions, we're going to submit and proceed to the Population page.

Once we do that you'll see, if you look back at the left-hand navigation bar, we've moved to Section Three here, which is the review and edit program features and data. Basically, in this section, you have the option of entering data specific to your population and your program, or you have the option of accepting the data the Calculator provides.

What we're going to do right now is review the default data. I'm going to tell you a bit about where the data that you see come from. Then, when we go through it a second time, we'll enter some specific data. What you see now are data about the Alabama Medicaid population. Once we enter Alabama and Medicaid on that Required Information page, the Calculator has the size of that third-party population and it also has data on the age, gender, and race breakdown of that population. These data are all resident in the Calculator and come from the Centers for Medicare and Medicaid.

We've looked at these data, and they look o.k. to us. So we're going to hit Submit and proceed to the Participants page. Before I do that, let me just note that if you had chosen private health insurance or you're not focusing on the Medicaid population, the Calculator uses other data sources like census data to populate this page. Now we're submitting and proceeding to the Participants page. On the Participants page, we're able to confirm or change the Calculator's estimate of the number of eligible participants in the program. And the Calculator generates the estimate, which you see here, right here as notations, eligible for the program. It generates this estimate by applying asthma prevalence rates for the population data we just saw from a preceding page. So the asthma prevalence rates are resident behind the scenes in the Calculator. And those rates come from Market Scan, Medicaid, and commercial claims databases.

In this case, since we're doing this exercise using Medicaid data, the Calculator is tapping into those Medicaid prevalence rates, which are broken down by age, gender, and in the case of Medicaid, race. It applies those accordingly. So the Calculator estimates in this example that there are about 69,000 patients eligible for an asthma care quality improvement program in Alabama. We don't expect that every single asthma patient who's eligible is going to participate, so we have that maybe 25 percent of these eligible patients will participate, giving us 17,000 patients who are expected to participate. So again, let's accept this and move on.

The next item for our review is the baseline data that Ginger and Rosanna discussed. This page shows us these asthma-related health care utilization costs associated with the population we defined. It tells us that for these asthma patients that we're focusing on - any asthma patient - on average they make .015 trips to the ED each year. On average a very small number of hospital stays because this is the general asthma population, and then on average about one outpatient visit per year. Again, these data are coming from the Market Scan database. These are specific to insurance types. If we had entered private health insurance we would see different baseline data. They're not State-specific, however, so during our second walk-through I'm going to show you what Alabama Medicaid has done and what you might do to enter specific data on this page.

We're going to move on and the next thing is the Program Impact page. What you see here are the results of the meta-analysis. This shows how a quality improvement program will potentially affect the costs and utilization that we saw on the preceding age. These data and these fields are going to vary, depending on some of the information that we entered on the Required Information page. We chose to look at studies that were randomized control studies. If we had made a different selection, we'd see different estimates in here. If we had chosen a different asthma severity level, we'd see different results in here.

I just want to point out at this time that if you scroll down the page over here on the left-hand side, you can be reminded of what you entered on the preceding pages. Sometimes you might be going through and say, "Which one did I select?" Here you can see the choices that you've made. Again, we chose all asthma severity level and we chose randomized control studies.

Again, we're not going to change this right now. As Ginger did say, this is all based on the literature. If you run your own pilot program and you know how your program affects these factors, then you can enter your own data here. So we're going to accept and move on.

The last step that we have is to confirm information about the setup of the quality improvement program. This is populated again with default data. This page asks about how long the program is going to operate, and you see the default is 5 years. For how many years until the program achieves full impact, we've got a default of 2 years. We have the annual program cost per participant set at $421, which as Ginger mentioned, is the inflation-adjusted average value from those studies. We have the discount rate for the ROI calculation. So we're going to leave everything as is and proceed to the Results page.

So here we are. And if you look back at this left-hand navigation bar, we've moved down into this Review Results section. There are two different ways to look at the results. You can look at the averages per participant or you can look at the totals for all the participants. We're just going to stick with looking at the averages.

The Calculator generates a couple of different things. There's the health care savings per participant. There's the total program cost per participant, productivity gain, and overall impact of the program. So here's where we have our ROI. In this case, you see that the health care savings per participant are pretty small. We've got $22, based on our hypothetical program that we created. And the productivity gains are more substantial, of course. The ROI we have over here is 89 cents, so that means that for every dollar invested in the program, 89 cents are returned. So basically every dollar of investment in this program is costing 11 cents.

Now what we're going to do is quit this and save this scenario to an Excel file. And we've developed this Calculator so that you can go through it a few times and save all your scenarios and then in the end you'll have an Excel file that shows you everything that you've input and then the results. And it's very easy to then take that and do some analyses with all the data you've generated.

Up here at the top, it tells us our scenario has been saved successfully. So I'm going to go back and walk through a second scenario. And then we'll open up the Excel file when we finish that second scenario and we'll take a look at everything.

We're going to go back to the Required Information page and, as I said, we're going to keep our focus on the Alabama Medicaid population. We're going to leave everything exactly the same on this page except for the severity level. We're going to select persistent asthma as our severity level. The reason is, as Stephanie Lindsay is going to describe, Alabama Medicaid has been implementing a pilot program, a pilot asthma quality improvement program. They're focusing on patients with persistent asthma, so they're focusing on a subset of all asthma patients.

We're going to submit and proceed to the Population page again. We're not going to change anything on this page. We're going to go straight to the Participants page. If you know how many participants you've got in your program, there's no need to enter the population data. That's really just used by the Calculator to generate the number of asthma patients eligible for the program in lieu of other information. But if you know how many you've got, you can just go ahead to this Participants page. Just move through the Population page and come to the Participants page. Where it says "Asthma Patients Eligible for Program," we're going to enter the number of potential patients, which in the case of Alabama's program is 21,000. And then, based on other experiences with the pilot program, about 5.7 percent of those 21,000 patients have been involved in the pilot program.

So this time I'm going to hit Submit and Review Changes, because the Calculator is going to just apply that 5.7 percent to the total eligible to come up with this 1,223 expected patients who will participate. And, as Stephanie will go into, that's approximately the number of patients they've had participating.

We're going to now submit and proceed to the Baseline page. This is another opportunity to enter your own data so that you can have the estimate be customized to your population and your situation. So the Alabama Medicaid team has gone through and, using those 21,000 patients they identified as being eligible for their program, they've run through all their claims and they've generated estimates of utilization and costs for their patients. We're going to replace the default data in the Calculator with the data that Alabama Medicaid has generated. So you can see as I'm doing this that the data that reside in the Calculator for the baseline cost and utilization data are from the Market Scan Medicaid data. They're not State specific, so this is great that Alabama has State-specific data here. I'm not going to replace the estimates pertaining to missed work days and missed school days; we're just going to leave them as is.

We can submit and review changes and then proceed to the Program Impact page. Again, we're not going to change anything here. You know, these are the results of the meta-analysis, so we're going to submit and proceed to the Program Cost page. We're going to leave this exactly as it is because I want to show you kind of the comparison between the first scenario and the second scenario by just changing a couple of things. It let me submit and proceed to the Results page.

And then we see the results. Again, the program cost was set up as a 5-year program, so you can see that based on the data - based on changing the group that we focused on from all asthma to persistent asthma and then changing the baseline data - we have a significantly higher health care savings here. And the costs are obviously going to be the same but the return on investment obviously is much, much better in this scenario. Again, we chose on the Required Information page to include both the health care savings and the productivity gains. When we're looking at the return on investment, we're looking at the combination of health care savings and productivity gains in relation to the program costs.

I'm going to save this scenario to an Excel file. It goes to the same Excel file the last one went to. Before I click away from this screen, I just want to remind you that on the left-hand side, as you're going through, this assumption bar continues to track the information you've entered. So that's just kind of a handy on-screen reminder of things. But certainly when you get to the end and you see the results you can then do this save into Excel and open the Excel file.

I'm going to open the Excel file and what you'll see is there's an input tab for each scenario you've run. There's an input tab that keeps track of all the choices you made on the Required Information page and the default data that went into the Calculator. And then there's a scenario tab. It's called scenario but this is a results tab essentially that shows the results of the scenario you ran. I ran this this morning and I took these two scenarios I ran and I just made a quick summary tab. Depending on what you're doing, as you're doing this it's helpful to have this Excel menu, as you can just look across things.

But just to show you the results and how they compare to each other: In the first scenario we looked at all asthma patients and in the second scenario we changed the definition to look at those with persistent asthma only. Of course, we entered our own data here for the participants but it does affect the program impact estimates that are applied. So it's important to change that, even if you're entering your own data, for everything. We see differences in the number of program participants. I just brought in the baseline data that we entered so that you can compare the differences. I realize that I'm running through this so you can't look at it too closely. But then as you get down here to the bottom, I just put the results right next to each other so you can see how, by focusing on a population that has higher utilization and costs associated with their asthma, you can have more of an impact.

I'm going to go and end my desktop sharing and I'm going to return to the PowerPoint. Just to summarize these steps I've gone over: You've got to fill in the required information piece. That's Section Two of the Calculator. That tells the Calculator what your type of population is and what studies you want to use in the calculation. It's very helpful as you're considering how to go about creating a program or considering a program or changing a program to remember that if you use the Calculator you'll see that targeting the most expensive patients generally pays off. Using the controlled studies helps you to achieve more realistic results as you're using the Calculator.

Then I just have listed here the steps the Calculator goes though. It estimates the population, and based on that it estimates the number of participants. You, at any point, can change these data that it's using to generate the final return on investment. If you're going to use your own data, you ideally want to find data from a single source. For example, the Alabama Medicaid team used the claims to determine the number of eligible asthma patients for their program. And then they used the same claims to determine the baseline utilization and cost data. So it worked out very nicely.

So what I want to do now is turn things over to Stephanie Lindsay, who's from the Alabama Medicaid agency. She is going to speak about their use of the Calculator in their evaluation of the pilot of their Q4U program, which is the care management component of Alabama's larger initiative, which is called Together for Quality. That initiative is focused on diabetes and asthma. Stephanie, I'm going to turn it to you now.

Stephanie Lindsay: Thank you, Elizabeth. Alabama Medicaid was introduced to several of the AHRQ tools at a conference held in January 2008 in Arizona. We were especially interested in the return-on-investment tools in order to determine a return on investment for new programs and because we had been instructed that all newly implemented programs should be budget-neutral. Our commissioner specifically noted that CHCS and AHRQ have return-on-investment calculators that we should review. Alabama Medicaid began implementation of the Together for Quality (T4Q) pilot program, a Medicaid transformation grant initiative, in 2008 in 11 counties.

There were three components to this initiative. They are Qtool, Qx, and Q4U. The Q4U portion of this program began in February 2008 in two counties with a phased implementation including 8 of the 11 counties in the T4Q pilot. Q4U focuses on the use of chronic care management for individuals identified by Alabama Medicaid with diabetes and asthma. We hope to expand this initiative within the coming year. With the help of AHRQ and Thomson Reuters, we have been able to take the data that we have obtained from our query system regarding the cost of Alabama Medicaid's asthma population and put this information into an application that can easily compute the savings or the cost of the program. We have worked with AHRQ to use that Asthma ROI Calculator in a way that will hopefully be beneficial to our program. We were able to assist Thomson Reuters and the AHRQ teams to find more user-friendly ways to disseminate the information, for example, the ability to save the data into an Excel spreadsheet. We were also able to work through a few initial issues with the Calculator.

We plan to use the data gathered from the tool to work toward identifying solutions for the recipients that we serve. For example, we can determine whether Q4U, as designed, is having a positive impact on ER visits and the value of that impact, and then look to see if changes could be made that might result in a greater impact. We are able to determine how much we are spending or saving per recipient and collectively for the asthma population. We will be able to make recommendations to our commissioner for changes to the program while identifying what the impact of this program will be, not always to Medicaid with the fiscal impact but also the overall impact, including productivity.

Thank you all for allowing me the opportunity to share Alabama Medicaid's experience with AHRQ and Thomson Reuters and the Asthma Return-on-Investment Calculator. Now we'll go back to Margie for questions.

Margie Shofer: Thanks, Stephanie. It is time again for a Q and A session and, as a quick reminder, we'd love to hear your verbal questions and if you'd like to do that please hit *1 and the operator will put you in the queue. Or, of course, you can ask questions via the Q and A tab on the right-hand toolbar on your screen beneath the participant list.

Question: Is the ROI Calculator free on the AHRQ Web site?

Margie Shofer: It is. And the last slide is going to direct you to the link so that you can access it.

Question: I wondered about the cost of those asthma programs and how you arrived at the cost for the programs?

Ginger Smith Carls: Thanks, that's a great question. Only a few of the studies actually published their cost for them to implement it. There were only seven, so we just took the average of those seven studies. Certainly that's something that uses the Calculator, so I'm probably going to want to get their own information on that, talking to a vendor or other people with experience on that particular program that they're planning on implementing. Because there were very limited data on that.

Question: My question is about the asthma severity. My guess was that you have categorized it as such based on what was available, but obviously this is not completely in line with the NAEPP guidelines insomuch as you're only looking at persistent asthma. But obviously under persistent there's no breakout of mild, moderate, severe. Is that something that you're able to elicit from the studies that were published and can add into the Calculator at a later time?

Ginger Smith Carls: I was involved in reviewing the literature. We did look at how finely we could break that out and, based on what those studies reported, it seemed like all you could really get out of it was those levels that we used. Because there were variations in the level of detail describing their populations. So that would be a great improvement. But certainly no literature through 2007 really supported that level of detail. There may be some new studies that fine-tune it more. The other piece of that, not just the literature, is that you want to collect information on what it costs to treat mild but persistent asthma. So we decided to use medical claims data. We figured other users of the Calculator might want to use something similar. I don't believe that you can get that fine a resolution from claims data. Maybe I'm wrong on that, maybe you know more about that, but I think it's just we use these data. The persistent asthma definition is based on the Healthcare Effectiveness Data and Information Set (HEDIS) definition and it's based on medication use and also utilization of hospital visits, ER visits, and any excessive outpatient visits.

Question: What are your recommendations on how to determine the impact of a program?

Rosanna Coffey: This is Rosanna and I assume you're talking about the impact of the program that you would implement. What you're getting out of the Calculator is an assessment of your population and perhaps your baseline utilization or cost information compared against what the literature would tell you is the average impact that you would expect for the particular one selected. So you're getting a view of what others have accomplished. And then what you would want to do is look at your own data, do a before/after and a control group and look at the change in all of those components that are in the Calculator and calculate a utilization change. Estimate the costs - as we've done in the Calculator - and come up with your own program impact for the program that you've implemented. So it requires more work than just going in and putting things into the Calculator because there you're seeing what other people have achieved. What you want to know is what your program has achieved. But you can follow the same methods to figure it out for your program.

Question: Please explain break-even program impact again in the Results section?

Ginger Smith Carls: There's a field called Break-Even Program Impact and that tells you how much you need to reduce - and I believe it's in percentage terms - the cost to get to break-even, meaning for every dollar you invest you get a dollar back. There's also a break-even program cost that tells you, given whatever I've put in for a pilot study, how much should my program cost so that every dollar that I invest in the program I'm going to get a dollar back in savings?

So that would be the number you'd want to take to your vendor and negotiate with to try to achieve.

Question: Can we use the ROI Calculator to estimate long-term impact after the program has ended? For example, if patients get rid of triggers for asthma or know how to manage their asthma for the rest of their life.

Rosanna Coffey: That's a good question. The studies that we have - and Ginger, you're going to want to jump in probably - the studies that we have for the most part are 2-year studies.

Ginger Smith Carls: They're mostly 1-year studies. There are a few that are longer, but longer term impact is not really well studied. It would be great to have that information but certainly in the literature we looked at, it's not in there. Maybe there's some more information out there since the enterprise, but there's not a whole lot of evidence on that, unfortunately.

Question: It still seems like cost of medications and other utilization numbers are critical. Where do we find these numbers: carriers, Medicaid, etc.? Where we collect this cost seems confusing with multiple carriers. Would this work for a moderate-size practice?

Rosanna Coffey: Again, this is for a practice and trying to find these numbers for your patients, I agree with you, is going to be difficult. I don't know whether you have any ideas, Ginger.

Ginger Smith Carls: I think the problem here is again, the practice is just one part of the system and this requires an integrated system to be able to look at costs from all these different perspectives, not just outpatient but inpatient. I don't know where you would get that. You could use what we have as an approximation. But you'd probably want something more custom if you were going to use it in some sort of negotiations.

I'm trying to think whether, if you specify your population more carefully and then go to a carrier, or even a Market Scan database, we could tease out estimates for use for a more careful definition of your population. But that's a synthetic estimate for you.

Question: How can you get estimates for the average ED visit per patient?

Rosanna Coffey: Again, it depends on what population you're looking at. For Medicaid, they certainly have that information; health insurance carriers, private insurance companies would have that information. That's where most of this information comes from.

Question: Will this presentation will be available for later review online?

Margie Shofer: Yes. We will have it up on the AHRQ Web site later but sometimes it takes us a little while to get these up, so please be patient. We probably will send out an E-mail letting people know when it is available.

I really want to thank everyone for the presentations. And I want to thank you, the audience, for your thoughtful questions and your participation in this Web conference today. We hope this discussion has been helpful to you. If you have any questions about the Calculator or if you have any technical assistance needs on how to use it, please don't hesitate to submit them to the quality tools E-mail address, which can be found on the slide. It is the third bullet down. Also it shows how to access the Calculator; that's the second bullet down. And, of course, you can always feel free to contact me, Margie Shofer. So again, if you have any questions or comments about this tool, please let us know. That concludes today's Web conference. Thank you.

Return to Contents

The information on this page is archived and provided for reference purposes only.


AHRQ Advancing Excellence in Health Care