Skip Navigation Archive: U.S. Department of Health and Human Services
Archive: Agency for Healthcare Research Quality
Archive print banner

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to for current information.

Transcript of Web Conference

Session 2: Safety Net Data Collection Strategies

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to for current information.

This Web Conference consisted of three sessions broadcast via the World Wide Web and telephone September 23, 24, and 25, 2003. It was designed to inform State and community officials about and teach them to use Data Books and tools for monitoring the health care safety net. This initiative consists of a broad range of local area measures related to safety net providers and the populations they serve. The User Liaison Program (ULP) of the Agency for Healthcare Research and Quality (AHRQ) developed and sponsored the program.

Cindy DiBiasi: Good afternoon. Welcome to Safety Net Data Collection Strategies. This is the second event in a series of three Web-assisted audio conferences on monitoring the healthcare safety nets. These events are designed for state and local health officials. The series is sponsored by the U.S. Department of Health and Human Services Agency for Healthcare Research and Quality, often referred to by the acronym AHRQ or AHRQ and the Health Resources and Services Administration or HRSA. My name is Cindy DiBiasi and I will be your moderator for today's session.

In 2000, the Institute of Medicine released a report describing the healthcare safety net as "in tact but in danger." The safety net, as you know, is the nation's system of providing healthcare to low income and other vulnerable populations. In particular, the report emphasizes the precarious financial situation of many institutions that provide care to Medicaid, uninsured, and other vulnerable patients. It also examines the changing financial, economic and social environment in which these institutions operate and it looks at the highly localized patchwork structure of the safety net. One of the five key recommendations in the report focused on the need for data systems and measures to assess the performance of the safety net and health outcomes of vulnerable population.

In response, AHRQ and HRSA are leading a joint safety net monitoring initiative. This initiative involves a three-part strategy focusing on both safety net providers and the populations they serve. As a result, they resolved to create two data books that describe baseline information on a wide variety of local safety nets, developed a tool kit for state and local policymakers, planners and analysts to assist them in monitoring the status of their local safety nets, and identify the data elements that would be needed to effectively monitor the capacity and performance of local safety nets. Information related to the AHRQ and HRSA initiative is available on AHRQ's Website at

As we talk about the safety net, it is important to make sure that there is a common understanding regarding what it encompasses. The healthcare safety net consists of a wide variety of providers delivering care to low income and other vulnerable populations. These include the uninsured and those covered by Medicaid. Many of these providers have either a legal mandate or an explicit policy to provide services regardless of a patient's ability to pay. Major safety net providers include public hospitals and community health centers, as well as teaching and community hospitals, private physicians, and other providers who deliver a substantial amount of care to these populations.

In addition to today's event, one more call will be conducted as part of this series. The next call, scheduled for tomorrow, Thursday, September 25th from 2-3:30 PM Eastern Time, will focus on how data will be used to tell the safety net story. I will tell you more about this call later in the broadcast, but right now let's turn to today's call.

Yesterday we discussed the new data books and the range of measures they provide for monitoring the status of local safety nets and the populations they serve. Today we will examine three chapters that are included in the forthcoming book entitled Monitoring the Healthcare Safety Nets, Book III, Tools for Monitoring the Healthcare Safety Nets. It will be available later this fall. We will tell our listeners how to get this new book at the end of today's discussion.

Similar to the data book, the tool kit is designed to help policy analysts and planners at the state and local levels assess the performance and needs of their local safety nets. Chapters included in this book are written by experts in the field covering a wide variety of topics and today we will examine three reports focused on data collection strategies.

Let me begin by introducing today's panelists. In the studio with me I have Lynn Blewett, assistant professor at the University of Minnesota, School of Public Health. Joel Cantor, director for the Center for State Health Policy of Rutgers University. Timothy Clouse, agricultural economist and statistician at the U.S. Department of Agriculture and Vickie Gates, senior consultant to the State Coverage Initiatives Program. Also here with us in the studio is Robin Weinick, senior research scientist and senior advisor on safety nets and low-income populations at AHRQ. As the lead of the AHRQ Safety Net Monitoring Initiative, she will be with us today in the studio and join us during the question and answer session. Welcome everyone.

Before we begin our discussion, I would like to tell the audience a bit about the format of this Web-assisted audio conference. First we will talk with our three panelists, then open up the lines to take your questions. We will give instructions on how to send your questions to us later in the program. In the meantime, if you experience any Web-related technical difficulties during this event, please click the "help" function in your window to trouble-shoot your Web connection. If it appears that the slides are not advancing, you may need to restart your browser and log on again. If you are on the phone, dial "*0" to be connected to technical assistance. Also, if you have difficulty with the audio stream or if you experience an uncomfortable lag time between the streamed audio and slide presentation, we encourage you to access the audio by your phone. The number is 1-888-469-5316. This is the same number to call to ask questions when we get to the question and answer portion of the program.

Now I think we are ready to tackle today's topic, Safety Net Data Collection Strategies, and Lynn Blewett, let's begin with you, assistant professor at the University of Minnesota School of Public Health. Lynn and her colleagues from the State Health Access Data Assistance Center at the University of Minnesota prepared a report for the safety net tool kit. It is entitled, Estimating the Size of the Uninsured and Other Vulnerable Populations in a Local Area. Lynn, what are the mechanisms to measure levels of insurance at the state and local levels?

Lynn Blewett: Well, the first mechanism is a survey tool to get a direct estimate of uninsured. You can either do that through a household survey where you ask people directly, do they have health insurance coverage or not or often researchers look at employer surveys to look at whether employers offer insurance and whether their employees take up insurance. Joel is going to talk a little bit about in the next presentation about how to do a local survey. The other thing you can do is develop either a proxy measure for uninsurance or a model-based estimate for uninsurance.

Cindy DiBiasi: Are there national data sources that provide state-level estimates of uninsurance rates?

Lynn Blewett: Yes, it is very important for local researchers and analysts to understand that there are national survey data that provide mostly state-level estimates, but some local estimates of uninsurance. Those are the current population surveys conducted by the Census Bureau. The Medical Expenditure Panel Survey, which is an employer survey, conducted by the Agency for Healthcare Research and Quality and there is a recent survey that is just being released this fall, the State and Local Area Integrated Telephone Survey or SLAITS that is conducted by the National Center for Health Statistics.

It is important to know that the current population survey does have estimates of uninsurance for the larger counties and metropolitan areas and you may be able to get information from the current population surveys.

Cindy DiBiasi: What about state surveys?

Lynn Blewett: Well, there are, because there is a lack of estimates at the local level, states have developed their own surveys to estimate health insurance coverage. We have documented at least 37 states who conduct their own household surveys. Many of these surveys are funded by the Health Resources Services Administration's State Planning Grant Program. The state surveys typically have larger sample size and targeted populations so you can actually get estimates at the regional and local county levels. So it is important that if your state has a survey, you should contact your state survey folks to see if they have estimates of your county or region.

Cindy DiBiasi: Now are there national surveys that provide information on health insurance coverage at the local level?

Lynn Blewett: There are a couple privately funded surveys that analysts should be aware of. There is the National Survey of American Families that is conducted by the Urban Institute. They have comprehensive survey data on health insurance coverage and income and poverty for 13 states. Those data are available in public use files available to state analysts.

The Community Tracking Study, conducted by the Center for Health System Change, also have comprehensive survey data on providers, households, and other survey information. Again, only for 12 communities. But if you are one of those communities, you ought to be aware that that data is available and the lists of those states and communities are in the paper, the Took Kit, Book III, which will be published later this fall.

Cindy DiBiasi: When direct measures aren't available, how can state and local analysts develop community-level estimates of uninsurance rates?

Lynn Blewett: Well, there are a couple of things you can do. One is to use an available measure to serve as a proxy for health insurance coverage. So while you may not have health insurance coverage, there may be a number of different variables that you could look at in your local area. For example, you could look at the self-paid variable from your hospital administrative records to estimate local areas of uninsurance. That self-paid variable is available in the data book that was talked about in yesterday's phone call. That would be used as a proxy for health insurance coverage when you don't have access to direct measures.

Another way is to develop a model-based approach, which is basically a statistical model that would predict health insurance coverage using different variables that are correlated with health insurance coverage. An example of this is looking at the state unemployment rate and levels of uninsurance at the state level and then applying this at the local level. So again, you use available data that you have to develop an estimate of uninsurance at the local levels.

Cindy DiBiasi: Let's talk about the strengths and weaknesses of the different approaches that you have mentioned such as estimating levels of uninsurance at the local level.

Lynn Blewett: Well, the best way to get a good, precise estimate of health insurance coverage is to conduct a survey with a large enough sample size. The downside of that is that it is very costly to do and what you really want to do is at least two points in time so you have information to look at change over time. Many local areas cannot afford to do a direct estimate or a household survey at the local level. Then you need to go and look at what available data you do have. The proxy measure is a pretty fairly easy way to get access to that information. It is low cost, data is available, but really you are not measuring uninsurance, you are looking at a proxy for uninsurance. So there may be some (unclear) in terms of what you are measuring.

The model-based approach has greater predictive accuracy than the proxy measure that is quite complex and you are likely will need some statistician or partner with your local university to help you develop a model that can be used. It could incur some costs in terms of getting the expertise needed to develop a good model-based estimate.

Cindy DiBiasi: Are there researchers working on these models?

Lynn Blewett: Yes, there are a couple of national federal agencies that are spending quite a few resources and expertise trying to develop local, basically county level estimates of uninsurance. The Census Bureau is working with current population surveys that currently provide state-level estimates and they are trying to develop a model-based estimate to look at county-level health insurance coverage estimates.

The Agency for Healthcare Research and Quality is also working with their employer survey, which I mentioned earlier, the MEPS employer survey, to look at local estimates of employer offer and take-up rates. Those will likely be available within the next year.

There are a number of states that have also developed models using their state survey data, including the State of Minnesota who has developed model-based estimates at the county level. The State of California has also developed estimates and there are other states that are working on their own model-based estimates.

Cindy DiBiasi: So keeping all of this in mind, what is your advice to people who are interested in trying to estimate levels of uninsurance?

Lynn Blewett: Well, my advice would be to just be knowledgeable and aware of the different data sources that are available for your communities. To use existing resources and data and build on your state and local survey activities. Many states have surveys and if you can add additional samples for your local area, you can get a lot of information just by piggybacking off ongoing activities.

It is also important if you are going to use a proxy or a model-based estimate that you use multiple approaches to look at the different estimates and try to benchmark against either the state or the national survey data. Also to be flexible as new data become available. The Federal Government is always looking at new ways to estimate levels of uninsurance. There is always new information available. You need to be aware of those activities and be flexible in understanding your local markets.

Cindy DiBiasi: OK Lynn, thank you. We will be back to you during the question and answer period. Now I would like to turn to Joel Cantor from the Center for State Health Policy at Rutgers University. Joel is the author of a chapter for the Safety Net Tool Kit entitled Local Data Collection Strategies for Safety Net Assessments. Joel, how should a local group decide whether to sponsor its own safety net assessment survey?

Joel Cantor: I am going to start really where Lynn left off. That is first and foremost to look at what is already available. It is very costly and very difficult to reinvent the wheel in this case. So if there are data available for your local level, go ahead and use it. Also look into the proxy method. Look into the small-area estimation methods. If those don't meet your needs, then you might need to do your own local survey. Local surveys can also be a useful community-organizing tool for local coalitions. They are useful for engaging stakeholders in their design and of course in going through the findings. Local surveys can also be extraordinarily useful if you have locale-specific questions to ask. If you want to know, for example, about the performance of individual providers in your community, the only way you are going to be able to ask those questions is to do your own survey.

Next you have to ask yourself, can the population in the survey actually report what I want to know? Household members of general populations, for instance, can't report for example the specialty of the physicians they see. But a survey of providers might be useful in understanding the availability of specialty care. Likewise, people can't necessarily report characteristics of, detailed characteristics of their employers. So in some instances employer surveys are necessary.

Finally, and perhaps most importantly, look at the resources you have available. Do you have the time to design a survey? Do you have the resources, the dollars available to collect the data, analyze the data and prepare the findings?

Cindy DiBiasi: Let's say my local access coalition decides to sponsor a survey. What is the first step?

Joel Cantor: The first step is to be absolutely clear about what your objectives are. You might want to focus on primary care. You might want to focus on specialty care. Satisfaction with services. There is a whole range of possible goals for a survey engaged with the local stakeholders. You could go through a consensus process to find what your objectives are. Those objectives should be written down and in essence serve as your manifesto for moving forward as you design and implement the survey.

Cindy DiBiasi: How can the survey objectives be translated into a questionnaire?

Joel Cantor: My first advice here is not to reinvent the wheel again. Use existing questions. There are ample surveys that have been developed that address healthcare access questions. Lynn mentioned several in her presentation a moment ago. Others, the Medical Expenditure Panel Survey, sponsored by AHRQ, are an excellent source of questions not only for general populations but also employers. There are others as well. I would also refer you to the SHADAC Website which has a wonderful bank of questions that can be adapted for local surveys.

But again, my first advice here is don't reinvent. Use existing resources and refer back to your manifesto, to your survey objectives, in selecting your questions.

Cindy DiBiasi: What kinds of questions do safety net access surveys typically ask?

Joel Cantor: Typically, they ask about health coverage, about health status, whether a specific condition such as diabetes or asthma or general perceived health status, various access utilization, usual source of care, attitudes and beliefs can be important. Characteristics of employers, for example whether or not they offer coverage. The size of an employer can be important. Wage levels, socioeconomic status, education, demographics, languages spoken and so forth.

Again, don't forget to refer back to your objectives and also don't forget to tailor your questions to the local circumstances, the local populations and the local questions you have, the local objectives you have for your survey.

Cindy DiBiasi: So besides the questionnaire, what other things do survey sponsors need to consider?

Joel Cantor: There are a myriad considerations in designing a survey. The sample size and the sampling strategy, issues of confidentiality, and how what promises will be made, and how you will follow through on those promises. Response rate issues, follow-up strategies. It is never enough to invite people to respond once. Unfortunately, people are hurried and busy and it takes a lot of effort, often multiple contacts, and sometimes a dozen contacts to get even a 50% response rate in some instances.

Interview mode, that is will it be a phone survey, in-person interviewing, through the mail are also questions. You will need to know whether the households in the area you would like to survey have a low telephone coverage rate. The Census Bureau has information on the proportion of households without phones. Increasingly, surveys of this kind are using mixed modes. They do a lot of phone interviews, but then they supplement them with in-person interviews in census tracts with low phone coverage. There are also some other techniques available for adjusting for a lack of phones in a household.

The next issue is selecting and training interviewers. Interviewing is a complicated skill that requires training and experienced interviewers are really invaluable. Then once you have your data, the story is not over. There are questions of data management, data editing, preparing analysis files, doing the statistical analysis. Even if it is not complicated, fancy regression analysis like Lynn was talking about, it is worth really thinking through what the analysis plan is and having the skills and talents to conduct the analysis.

Finally and absolutely not least, is reporting of findings. Too many times I have seen local groups conduct a survey, put a huge amount of resources and energy into collecting the data, then come to the end of the grant period or whatever it might be and the findings kind of trickle out. Put a huge amount of energy into reporting the findings. Briefings, papers, whatever means you want to use takes a lot of energy.

Cindy DiBiasi: OK, Joel, if we haven't scared everybody off by now (Laughs.).

Joel Cantor: That is my goal. (Laughs.)

Cindy DiBiasi: It sounds pretty complex. What are the biggest kinds of problems that novice survey sponsors should watch for?

Joel Cantor: Sure. First of all, remember why you are doing it. Work closely with the community that you are conducting the surveys for. Work with your stakeholders. Work with the population. Keep your eye on those objectives. Keep in mind why you are doing it, from beginning to end. It is very easy to get lost in the details.

The second pitfall is running out of money, time or just plain old running out of steam. As I said, very often, too often, survey sponsors get to the end and don't have the energy or the time to really get their findings out. So always really think about spending the time at the back end. But also think about spending the time in the development at the front end. Designing questionnaires, even if you are following my advice and drawing on existing questions takes a lot of time.

Next, pay attention to data quality, sampling problems. There is an instance in a group that I worked with in the past where they were really eager to get some quick data on the health status of the community so they used the local sanitarium and gave them clip boards and sent them out to the local malls to do interviews. Well none of them spoke Spanish and there were a lot of folks in this community who spoke Spanish. It wasn't a scientific sampling strategy so they really got quite a biased view of healthy kids visiting the mall.

Also, the next pitfall is losing your focus. Again, write down those objectives and stick to them. Then lastly, as I have said, really save enough time and energy for the analysis and reporting.

Cindy DiBiasi: OK, Joel, we will come back to you during the question and answer. Let's now turn to Timothy Clouse, an agricultural economist and statistician at the U.S. Department of Agriculture. Tim wrote the chapter Assessing Safety Net Provider Financial Help, A Simple Measurement Tool, in conjunction with two colleagues from the Health Resources and Services Administration. Tim, how do you assess what is good financial help for a community-based healthcare organization?

Timothy Clouse: Well, we ran into this problem when I was with HRSA in the southeast regional office. We have 150-odd community-funded health centers. Unfortunately, there is no way to say is this a good clinic or a bad clinic? It was kind of a subjective process. It was a matter of "Well, I know it when I see it," said in another context. Well, the problem arises in that it depends on who the observer is. So what we had to do was come up with a way of coming up with a sort of objective, verifiable, independent mechanism for saying this is or is not good financial help.

Cindy DiBiasi: So how does the measurement tool that you developed actually work?

Timothy Clouse: Well, in essence we decided that we wanted everybody to be above average. It doesn't make a whole lot of sense at the outset, but work with me on this. What we had initially was a sample, national sample of community-based health centers and we decided as the (unclear) there is no objective way of saying given this set of financial measurements, this is good, this is bad, this is indifferent. So instead what we went with was a sort of a relative ranking system. We knew we couldn't establish what they were objectively, but we did know that well, this clinic is probably worse than this other one. That is what we went with. We went with a sample. We selected a series of variables, financial measures that we thought were important and were closely related to them, and were not closely related to one another, actually measuring different things.

Cindy DiBiasi: What kind of data is required for the Financial Risk Assessment Worksheet?

Timothy Clouse: Well, it uses what should be readily available financial data, which is things that show up in a community-based healthcare organization's financial data things like a CPA report. So it would have things like current assets, current liabilities, cash on hand, things of that nature. When you take that and you plug it into this simple worksheet, well, relatively simple. It works sort of like the 1040 for your IRS paperwork. It gives you numbers and...

Cindy DiBiasi: Let's talk exactly about what that worksheet looks like.

Timothy Clouse: OK. Here is part of it. This is both in the document and there is an XLS version that you can download off the Internet, the HRQ Website. This is like the first page of it, which has got the year. One of the crucial things is, instructions is if you don't have it, assume, we allow you to assume the best. However, one important thing which doesn't show up here which is on the worksheet and in the document. This is not a substitute for a professional judgment. This is not a cookbook. All this does is give you an indication of where the issues are.

Cindy DiBiasi: Now let's talk about how this tool can identify problems.

Timothy Clouse: OK. There are a couple of ways it works. As you can see on the spreadsheet here, it shows some of the measures we have got to fill those out. When you run through the spreadsheet, it gives you an indication of where the problem areas are. We have actually used this. For example, there is one case in the southeast where we ran the model and looked at it. Turned out that this particular organization may have had some problems. We did some additional work with it, talked to them, and helped them out a little bit. It turned out that the issue was that their fee structure hadn't changed in several years. So we took that and said OK, based on the model and the additional work we have done, you need to look at what your fee structure is. They went back in and said, "Oh, well we haven't done anything with this for a while." So they revised it and the result was their financial health improved dramatically after that.

Cindy DiBiasi: So you could adopt it to local use.

Timothy Clouse: Yes, you can. That is one thing, one of the important things about it is once again, and it is situational. It is not a cookbook. If you know, based on what is going on in your community, that particular measure is not as important or more important than what you actually think it is, then you can adjust it. You can adjust the weights, as in the relative importance of each measure. If you think, for example, we have got the depreciation in there and we don't think it is very important, well, if the local community thinks that the age of the building is really important, they can make that, they can increase the importance of that measure.

Cindy DiBiasi: Change the weighting of it.

Timothy Clouse: Yes.

Cindy DiBiasi: How else can the worksheet be used?

Timothy Clouse: Well, actually you can also kind of use it backwards in the sense that you can use it to set up the community-based healthcare organization using these measures to say, "Well, if we want this organization to be viable", what you can do is say, "Well, we need to be able to pay off our bills within this particular time period. We need to collect from our insurance company within this particular time period. We need to have this dollar amount available on hand stashed to pay off any unexpected bills." You can also use it to, once you have got the organization up and running, the plan on how to keep it going over the future.

Cindy DiBiasi: OK. Timothy, we will be back to you. In a moment we are going to open up the discussion for questions from our listening audience. But first let's go to Vickie Gates. As a senior consultant for the State Coverage Initiatives Program funded by the Robert Wood Johnson Foundation, Vickie has seen various state and local safety net initiatives first hand and she has also worked extensively on safety net issues during her tenure as a state official in Oregon. Vickie, we heard lots of discussion today about data and surveys. Based on your years of experience, do these data and survey efforts really make a difference?

Vickie Gates: Well, I would never argue that there are not a lot of decisions made without the benefit of data, but when it comes to big investments today, I think increasingly people are not only asking for data, but they are also asking about the quality of that data. One of the things particularly as we have watched the HRSA State Planning Grant states over the last three years, Lynn, Joel and I all have seen situations where policymakers have literally changed the way they understood a problem and changed their position on an issue because of the data that was made available.

Now I really want to reiterate a point that both speakers have made. Data is expensive. One of the values I think of this work that we have seen today is that it is realistic. It really puts you with what is already a base to the information and if you are thinking about a survey, read the two papers from Joel and from Lynn. It is going to help you make sure that in fact your results are not going to be disappointing, that you are going to be clear about the objectives. You are going to have realistic objectives in terms of what data actually is going to be able to do for you. I think it is an incredibly valuable starting point.

The last point that I think I really want to reiterate from the speakers is this kind of what I would call "the softer side of data." It is that ability to use these processes to bring consensus to a situation, to put everybody on the same diagnostic level and in many cases actually put an issue on the policy agenda.

Cindy DiBiasi: Now you (unclear) as we said and most recently have collaborated with many states on solutions for the uninsured. Is a tool for assessing safety net provider health really of interest to state policymakers?

Vickie Gates: I think the answer to that is yes. I think it actually benefits the state policymakers for quite a period of time. There is no question. All states agree there is a need for the safety net. The difficulty is understanding what its actual status is and then trying to figure out once you have some way of knowing what is going on with that system, what should the role of the state be? How should the state be responding? In most cases, states are dealing with (unclear) when it comes to the system.

So what this may offer is a set of benchmarks in a way of looking at financial health. States do many things that have a major influence on the health of the safety net and their financial viability. They need a way to understand the possible implications. They need a way to understand what follows, what may actually may happen once they have made a change.

I think it is also this ability to benchmark for the clinics themselves to look across the state, to have that consistency. For policymakers at all levels, all of whom are concerned. The clinics themselves, the local government who is often a major funder, and say this could be a very valuable piece of the picture.

Cindy DiBiasi: What other opportunities do these new tools create for state and local policymakers?

Vickie Gates: Well, I think that at this time we may be getting enough of a picture to say we can talk about both sides of the equation, that we can talk about supply side issues and we can talk about demand issues and we can have good data and an ability to have some common understanding and benchmarks on both of them. As that ability to produce a credible picture that people can agree on, I think is one of the first steps for communities and states to find an effective way to work together in preserving access and expanding access to low income and uninsureds.

There is another point in the discussion that I also think is worth reiterating. Point in time investments are not the whole story. We really have to find more ways of monitoring over a period of time on both these issues. That is where a lot of the policy (unclear) is going to come.

The safety net has been characterized as fragile. I think that if you look today you will see states making really significant decisions about coverage. Trying to understand what the implication is for uninsured and for safety nets. This ability to turn to these tool kits I think is going to make it easier for states to feel at least knowledgeable about the social impact of these really difficult decisions in very hard financial times.

Proceed to Next Section

The information on this page is archived and provided for reference purposes only.

AHRQ Advancing Excellence in Health Care