This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: https://info.ahrq.gov. Let us know the nature of the problem, the Web address of what you want, and your contact information.
Please go to www.ahrq.gov for current information.
On September 9, 2009, Rosanna Coffey, Ginger Carls, and Elizabeth
Stranges presented a Web conference on the Asthma Return-on-Investment
(ROI) Calculator. This is the transcript of the event's presentation.
Select to access the conference
Margie Shofer: Hello, I'm Margie Shofer, with the Office of Communications and Knowledge Transfer at the Agency for Healthcare Research and Quality, also known as "AHRQ." Thank you for joining us for this for this Web conference on the Asthma Return-on-Investment, or ROI, Calculator. The Asthma ROI Calculator can help State officials decide whether it's cost-effective to establish an asthma care management program with online and evidence-based tools, and makes it easy for users to determine the cost savings that will likely be achieved by having an asthma care management program for a specific population, the areas to target so that asthma interventions can achieve cost savings, and the number of years the program must be in place to have a positive return-on-investment.
This Web conference is the third of a series of events highlighting the latest releases of several AHRQ tools. Earlier versions of these tools, including an earlier version of the Asthma ROI Calculator, were shared at two workshops for State policymakers and data analysts in December 2007 and January 2008. This Web conference series is a continuation of the technical assistance opportunities that have grown out of those initial workshops.
During today's event, we will spend time reviewing the Asthma ROI Calculator by conducting an overview of the Calculator, discussing the ways to use the Calculator, and examining the literature that supports the tool's findings. We will also touch on the some of the caveats that stem from the tool, and then we will have the tool developers conduct a step-by-step demonstration of the Calculator. We will take some questions after the background information about the tool has been presented and before the live demonstration. After the demonstration of the tool, we will move on to the Web conference's next component, a brief discussion of how Alabama's Medicaid Department has used the tool as a component of their evaluation of a pilot asthma quality improvement program. Finally, we will have a second open question and answer period in which we will address as many of your questions as possible.
We would appreciate your active participation, as the primary purposes of today's Web conference are to introduce you to the Asthma ROI Calculator, to get your feedback on the tool, explore practical applications for the tool, and learn how you envision using the tool and for what purpose. Related to this last point, we hope you will tell us about the types of technical assistance that you might need in order to make full or better use of the Asthma ROI Calculator.
Today's Web conference will include presentations by Rosanna Coffey, Ginger Smith Carls, Elizabeth Stranges, and Stephanie Lindsay. Rosanna Coffey is a Vice President in the Research Division of Thomson Reuters Healthcare. Recently she has been architect of the NHQR State Snapshots, the AHRQ Improvement Guides for Diabetes and Asthma, and coarchitect of this tool, the Asthma ROI Calculator. Ginger Carls is an economist in the Health and Productivity Research Department at Thomson Reuters Healthcare. At Thomson Reuters, Ms. Carls has worked on projects to evaluate the performance of disease management programs and cost/burden studies. Elizabeth Stranges is an analytic consultant at Thomson Reuters Healthcare, where she works on health care costs and utilization project reports with Substance Abuse and Mental Health Services Administration estimates of national expenditures on substance abuse and mental health treatment, and special studies pertaining to the National Health Care Quality Report and the National Health Care Disparities Report. Stephanie Lindsay, who will speak briefly about Alabama's use of the Asthma ROI Calculator, is a Medicaid Administrator for the Alabama Medicaid Agency. She has worked with Medicaid for 4 years, including 2 years in the Pharmacy Services Division, and is currently in the Statistical Analysis Unit.
As I mentioned a bit ago, this Web conference will have two question and answer periods, one before the live demonstration, and the second one after all the presentations are complete. There are two ways you can ask questions of our presenters today. You may submit a question at any time throughout this Web conference by typing your inquiry into the Q and A box located on the right-hand toolbar on your screen beneath the participant list. These questions will be entered into the queue and answered during the question and answer period, so feel free to submit them at any point during the presentation. Or during the question and answer period, you can ask your question verbally.
In order to be placed into the queue for verbal questions, please hit *1 on your phone. This will alert the event coordinator to your wish to ask a question and your line will be opened in turn. As those of you who have participated in the earlier Web conferences of this series will know, while we encourage questions regardless of their format, asking a verbal question can be easier if you have any followup questions or if the presenters need clarification from you. We will try our best to get to all questions but if we do not address your question due to heavy volume, please feel free to E-mail us and that E-mail address will be on the final slide.
So without further ado, I'm now going to turn this over to Rosanna and her team.
Rosanna Coffey: Hello, everyone. We're delighted that you have joined us today for this overview and demonstration of the Asthma Return-on-Investment Calculator. Margie actually went through this outline, but let me just restate it shortly. We're going to do an overview of the Calculator and how it's been used. I'll do that part. Then I'm going to turn it over to Ginger Carls, who is really the brain behind the Calculator. She did all the meta-analysis and helped with the development of the original Calculator. She will do the methods behind the Calculator and some things you should consider in using it, the caveats. We'll stop for questions and answers there. And then Elizabeth Stranges, who's been helping with the HTML version of the Calculator, will actually walk us through the Calculator and give us some real-life experience with how to use it. And then Stephanie Lindsay will give us insights from Alabama Medicaid and we'll follow with questions.
We would like to start with getting a sense of how many of you in the audience have actually reviewed the Calculator or used it already, maybe in its former incarnation. On the right side of your screen, you will see that there is a poll: "Have you ever used or reviewed the Asthma ROI Calculator?" If you'll take just a second to respond to that, we'll get the results momentarily. In the meantime, I'm going to start with the background information on the Calculator.
So why even bother developing a Calculator like this? AHRQ's purpose in doing this is to really help State policymakers, as well as health plans, estimate the potential financial returns from instituting an asthma quality improvement program with a specific population. If you go to the literature, you'll find that most of the research studies out there don't address the financial impact. Most of them only speak to clinical results and utilization impact. So the utilization impact must be converted to costs or savings for you to get some assessment of the financial returns that you would get from instituting a quality improvement program.
These estimates - this is a very high level view here - are developed from the literature, a meta-analysis, which is a study of studies basically, combining information on results from those studies with refined use and cost data that we have put into the Calculator. You have an opportunity to enter your own population information and other choices in the Calculator. You can make choices and enter other estimates as well to streamline it and hone it for your particular situation. And we're going to go to the AHRQ Web site and you're going to see that when we get to the demo.
How the Calculator works: you basically start with baseline utilization data. Say that you have 100,000 patients and you know what their ED visit rate per year is. That tells you how many annual visits you have per year. Then, from the literature, you know you could actually have an effect of reducing ED visits by 50 percent. That translates into cutting your ED visits in half. Then we assign the dollar cost of an ED visit. (And these numbers here are hypothetical numbers. I have made these up, so don't use them for some other purpose.) You then can calculate the amount of money you would have saved by instituting this program.
We would repeat this for all of the other components in the Calculator and we also add some productivity gains. Then you can find out after you've made an estimate of your program cost the cost per participant and multiply that times your number of participants. You simply take the dollars saved, the total dollars saved, divided by the total cost of the program, and you get your return on investment. Here it's a 75 cent return on investment, which means for every dollar you invest in this you get 75 cents back, or it costs you 25 cents per participant. The break-even return on investment would be a dollar.
So that's the big picture of how this works. We have done these calculations, not only for emergency room visits, but also hospital stays, outpatient visits, medication costs, and ancillary testing. And, as I mentioned, missed school or work days are where we look at productivity and we cost those out and then add the cost of implementing the program. Though the ways that you can use the Calculator are for planning a program, you could actually use it to forecast the financial impact of that program. We're going to walk through some of that later. You can look at the methods behind this and assess some key assumptions about a program you've proposed by thinking about "should I be targeting this population," or "maybe it shouldn't be adults and children, maybe it should just be children," and so on. You can make those kinds of judgments in the Calculator, enter and select some of those different target populations and see what happens to the impact. The other thing you can do is determine whether the assumptions that you've made about your own program are reasonable compared to what's found in the literature.
Another way to look at this is to say you have a program in effect and you don't think you're really saving money and you need to negotiate with your vendor on the cost side of the program. This helps you to figure out what you should be able to save and maybe to make some strong negotiations with your vendor and just strengthen your position with them. Then, of course, you can just look at the things that we looked at and that the literature looks at in terms of a return on investment. And that can help you plan an evaluation of an asthma care program.
These are some of the State groups and agencies that have already looked at the Calculator. We started this work with the Michigan Pediatric Asthma Coalition and they had a strong hand in what we did. They actually used this for making a funding decision for a county asthma program in Michigan. The New York Department of Health used it to talk about the evaluation components that they should be incorporating into a project where they were trying to change housing conditions and reduce asthma systems. Massachusetts, likewise, was looking at an environmental approach to asthma prevention and control, and they looked at the Calculator as well. We're going to talk more about Alabama because theirs is a really interesting application. They're planning a program and they're using the Calculator to plan and justify a pilot asthma care program for their Medicaid recipients.
Now I'm going to turn this over to Ginger Smith Carls, who's going to go through what's behind the Calculator.
Ginger Smith Carls: Thanks, Rosanna. Rosanna gave a broad overview of the Calculator and how you can use it. I'm going to dig a little deeper and talk about the whole foundation of it. We've done a literature review and put together a meta-analysis, as Rosanna said, which is basically a statistical study of studies. The reason we did this, as Rosanna mentioned, is that with a lot of studies it's hard to make sense of them just reading one after the other. This is a statistical way to make sense of them, not just reading one after the other. So I'm going to talk about what those studies are, but also keep in mind this just gives you the average results that other people have found. Certainly an actual program may do better or worse. And if you have your pilot study, you may want to plug in that information. When we get to the demonstration, you'll be able to see how you can choose to use the default literature review that we have in there or you can input your own information.
Moving on to the method, the literature review inclusion criteria, this just gives a description of the studies that we did include. We included children and adults under 65, we excluded the over-65 population. The rationale there was that for adults over 65 with asthma, some people also have COPD, chronic obstructive pulmonary disease. That can be a very different situation when you're treating asthma, so we wanted to keep everything just with asthma. All the studies were in the U.S. and the intervention involved self-management of asthma and sometimes a component of physician education. We excluded clinical trials and some very unusual interventions like a summer camp. The type of study: they were all individual interventions. The timeframe: we all had to follow patients for at least 6 months. So they excluded any of those studies that looked at very short-term effects. Most of the studies were for a year. There were a few that followed the patients for 2 years or longer.
We extracted studies that were published between 1995 and March 2007. More recent studies are not included in what we have done. But certainly if there's a study that you've seen more recently, you can use that information to put in the Calculator. If you'd like to find all the studies, we have some documentation where we have abstracts and the reference studies that are included.
Moving to the next slide, this table gives an overview of what studies there are for each of the different target populations along the top for the age groups, children only, adults only, both adults and children. Along the rows are the different aspects of asthma care: medical care, emergency department visits, hospitalizations, outpatient visits, medication costs, and ancillary services such as testing, for example, allergy testing. For emergency department visits, we had 44 different studies for all the different age groups and almost 23,000 patients. The hospitalizations had fewer studies, 33 studies, but those studies tend to be larger, so there actually end up being more patients.
The other thing to take way is that emergency department visits, hospitalizations, and outpatient visits have the most evidence. There's less evidence behind the productivity, missed work and school days, medication costs, and ancillary service costs. That's one thing to kind of keep in mind. If medication cost is going to be a really important part of your study, you might want to look carefully at the studies that are included. If it's a really important part of your intervention, you might want to look carefully to what's included to make sure that those are going to be relevant to what you're doing. We didn't really exclude; these are pretty much all the studies that were published on these kinds of self-management type of interventions.
To give you a little more detail on what a meta-analysis is, we had one observation in the data that was a study outcome. A study can have multiple outcomes if they reported both emergency room visits and hospitalizations. That would be two. If they reported them separately for children and adults, you could have additional outcomes. We did a regression analysis, so the dependent variable was the percentage change in hospitalizations, for example, and then the independent variables that were related to those outcomes were all these things listed: the age of the patients in the studies, if they were on Medicaid, the severity of asthma, and whether they had persistent asthma or intermittent asthma. Persistent asthma patients say they more frequently have asthma attacks and they typically are on controller medications. The study design used a randomized study where they used a statistical control group or they used no control group. If they have an intervention only focused on physicians, there will be controls for that. And then how long; if the outcome was measured after 2 years or longer, we also controlled for that, and then the sample size and the sample size squared.
Things to take away from what we've done with the meta-analysis is that it summarizes the literature along these different dimensions in terms of the average effects in the literature by age, Medicaid status, severity of asthma, and study design. If it was the standard design of randomized control trials versus statistical controls or a non-controlled study.
So that's the general takeaway. The average effect found in the literature for each of these different groups was taken from the meta-analysis and was put in the Calculator. When you know your target population, such as children with persistent asthma, it pulls up the average outcomes found by those studies and averaged through this meta-analysis process.
To continue to give you a little more information, this highlights what I had mentioned earlier about the literature. This list identifies which part of the literature has a lot of information and what we can have more confidence in and which parts of the literature have less information. We had 76 studies that met the inclusion criteria, and 52 of them had usable findings in a sense that they reported findings in a way we could use. Some studies don't always report all the information you need. So again, a lot of information on hospitalizations, emergency department visits, and outpatient visits, and a little bit less (there were 17 findings) on medication costs. One study might have contributed two different things. And you have very little information on the cost of ancillary services and the total indirect cost.
What I just completed is the information about the meta-analysis that provides the evidence base. Then the Calculator: what we have done when we were developing the Calculator is to populate the Calculator with a lot of baseline information that you can use if you don't have your own information. But if you have your own, you can put in your own information, which is probably the better way to do it or the preferred way to do it. So the Calculator includes baseline estimates for costs and utilization specific to the population selected. You can select the whole Nation or an individual State, Medicaid, employer-sponsored health insurance like private insurance, and State government employees and dependents. And, like I said, Elizabeth's demonstration is going to point out each of the different points where some of that information is provided.
In the last few slides, I'm just going to talk a little bit about some of the limitations or caveats about the evidence that's built in there that you want to keep in mind. So we mentioned the loaded evidence on some components. Only seven studies reported the program cost, which is the cost to administer the program, such as hiring the nurses to give the education or to send a mailing to the people with asthma. Very few studies reported that. We found an average of $395 per patient per year. So that's put in the Calculator as a default. I believe inflation adjusted that number. But there's a huge range, so the lowest was $81 per patient per year for an automated general educational mailing, all the way up to almost $1,000 for a really intensive intervention for the very highest cost asthma patients. Also, a few studies reported information about the impact of an asthma program on asthma medication costs. But in general, studies without a control group reported larger increases in medication costs associated with doing an asthma intervention. So the gold standard studies, the randomized control studies, generally found a smaller increase in medication costs. Another note about medication costs is that for the different studies who did evaluate the effect of their program on medication costs, the baseline - what each of the patients was spending at the start - varied a lot. That's another thing to keep in mind.
One concern is that the literature continues to grow, so, as we said, we monitor the literature through 2007. You can search the literature after April 2007 and more recently and take some of those studies and input their evidence.
Another concern is with baseline data that we have input. We have inflation-adjusted it but medical care and technology change, so baseline data do maybe become obsolete. You might want to consider using your own data from your own programs to populate the Calculator in terms of the prevalence of asthma, how much it costs to treat an ER visit, those types of things. And we had inflation-adjusted the data to 2008, but suppose you want to look at 2009 data or you want to project cost to 2009 or 2010. There's information on inflation factors described in the appendices that you can use if you need to be doing those things for your projections.
That's the end of my portion on the methods behind the Calculator: the literature review and then some of the baseline data, giving you some things to think about when you're using that information for your own purposes. Now I'm going to turn things over to Margie Shofer, who's going to lead the Q and A session.
Margie Shofer: We're at our first Q and A period. As I mentioned earlier, there are two ways you can ask questions. You can type your question into the Q and A tab located on the right-hand toolbar of your screen beneath the participant list or you can ask a question verbally by hitting *1 and then the operator will put you into the queue to ask questions. It would be really great to hear your questions verbally. It's very helpful and very interactive for us. So while folks are queuing up for verbal questions, I will take a written question.
Question: The first question is: How might the ROI Calculator allow an asthma practice to document higher quality of care for asthma patients?
Rosanna Coffey: This is a very interesting question because you're trying to get down to the individual physician or at least the clinic level in terms of asthma practice. And the ROI Calculator is designed for a look at a population. So if you have a substantial population that you're tracking, you could certainly look at the Calculator, play with it, see all the components that I mentioned early in the slide set, and evaluate those for your particular population. If you have a clinic that can poll information on people with asthma, we have in the Calculator definitions of what persistent asthma is for this purpose and persistent asthma with acute events. If you can go back into your population and calculate those things, you can enter those and evaluate where you have room to improve.
In terms of delivering higher quality of care for asthma patients, certainly you know that the National Asthma Education and Prevention Program (NAEPP) is basically what most of the literature followed in setting up asthma care improvement programs. So that's what you're being measured against in this Calculator. If your clinic already subscribes to all of those activities - the written asthma plan and education for the patient and encouraging the patient to take responsibility for managing the asthma and the controller medications and all of those things - then maybe you're already there. But by looking at all of the information behind the Calculator, you would maybe be able to figure that out in terms of where you need to improve.
Question: What size of population do you need to be able to use this Calculator? Are we talking 10,000 patients or do you need bigger numbers?
Rosanna Coffey: No, you can use any size population in this Calculator. It just doesn't make sense to do 1 or 2 patients or 10 patients because you're going to have such variability probably in small numbers in the estimates that you come up with. But there's no limit on the size of your population.
Question: The major focus of this Asthma Calculator seems to be hospitalizations and ER visits.
Rosanna Coffey: It's also outpatient visits, medication costs, and ancillary testing. All of those components are in there.
Question: In a private practice allergy and asthma group, we aren't privy to the number of primary care or ER visits they have had. We would have to work with a big insurance company to access the data. Are the insurance companies willing to provide the data to a smaller allergy and asthma practice?
Rosanna Coffey: That's a very good question. I can't answer it for the insurance plans that you're working with. I would think that they would be interested in improving the care for the patients that they're insuring, but I can't really know what they would do. I appreciate what you're saying. You don't know the number of ER visits; you don't know the total number of outpatient visits. But you could look at the baseline data and try to look at how your population compares for the thing that you do know - medication costs perhaps - but you may even have trouble with that. So the Calculator was really built for a big population kind of view. And trying to decide, you know, even on a State level, whether or not to undertake and invest in putting people into some kind of a care management program.
Question: Do the Medicaid data include SCHIP?
Ginger Smith Carls: The Medicaid data include several States. I think it's about 8 to 10 and those States, exactly how they're organized, the way they cover children, I'm not sure but it includes all their populations. So basically, we got our baseline cost and utilization data from those 8 to 10 Medicaid States. For prevalence, we have that from some mix from Medicaid services, so that would include the SCHIP programs. Does that answer the question?
Rosanna Coffey: Ginger, are you saying that SCHIP was included in some of the Medicaid States that we have and maybe not all? And that's something we would have to look up to know?
Ginger Smith Carls: I think we have to look up to know for sure. We didn't exclude anyone, so whoever is covered by their State's Medicaid program would be in there.
Question: I have two questions about the meta-analysis. One asks if it's published and then someone else asked if they could see somewhere the list of the 50 studies that were used. In what kind of settings were these done, and how does the intensity of the intervention vary in different studies?
Rosanna Coffey: To the first question, the study has not been published. The answer to the second is that we can provide a list. We have documentation on the meta-analysis, the information on all the studies and their abstracts. That information is available. At the end, last slide, there will be some E-mail addresses and you can E-mail someone at Thomson Reuters, even myself, Ginger Carls, or someone else, and we can provide that information to you.
Question: How can I find out about the asthma programs used?
Rosanna Coffey: That's also in our documentation and the abstracts of the studies will give you some information about that. We have that readily available. But if you want more detail, you have to go back to the actual studies.
Question: Were acuity levels considered in the ROI Calculator methodology - self-management versus clinical touch?
Rosanna Coffey: I'm not sure what you mean by self-management versus clinical touch. But the ROI Calculator distinguishes between anyone with asthma and people with persistent asthma and an acute visit in the year, which means they had a hospital stay for asthma or an emergency department visit for asthma.
The studies were classified based on the classification of severity in terms of how they described their sample and also based on their data. If they recruited everyone in their study from the ER, then they were in the persistent asthma with an acute visit category. A lot of the studies described their population as everyone with asthma and this is a wide range of severity. Or they said they were looking for the highest cost cases or the more persistent, more complicated cases.
Question: Does the Calculator offer a 95 percent confidence interval around the ROI estimates that it gives. Ginger?
Ginger Smith Carls: No, it does not.
Question: For medication costs, is it that it's calculating medication costs to the patients?
Rosanna Coffey: I think what they're asking is, there are multiple payers for medication costs. There are the patients paying their copays or deductibles, there's the insurance possibly, or if it's Medicaid, then Medicaid pretty much pays for all of it. The way we have set up the Calculator, you can select all the payments made or just the payments made by Medicaid or a government employer or the insurance company. You could look at it from a cost perspective. So that's a great question.
Question: Can the ROI Calculator be refined to a stratification level, high, medium, low level of asthma severity and from the stratification help to identify the most cost-effective approach to asthma management?
Rosanna Coffey: The answer is yes. And you're going to see that shortly when we do the demonstration.
Return to Contents
Proceed to Next Section