Skip Navigation U.S. Department of Health and Human Services www.hhs.gov
Agency for Healthcare Research Quality www.ahrq.gov
Archive print banner

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: https://info.ahrq.gov. Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to www.ahrq.gov for current information.

Local Data Collection Strategies for Safety Net Assessment (Continued)

Who to Survey

The first major decision in survey development is who to survey. As discussed, carefully defined survey objectives should be used in determining the survey target population. The target population, sometimes called the survey universe, is the group a survey is seeking to represent. For example, a target population might be all persons living in a local community or it might be low-income or uninsured persons in the community. It is vital to clearly identify this population, including the geography (e.g., within specified city limits) and population characteristics (e.g., specific age groups, insurance status, and income ranges).

Survey Sampling Strategies

Once the survey target population or universe is clearly and specifically identified, the next step is to identify possible sample frames and to consider sampling methods. A sample frame is a list of all members of the target population and ideally includes information needed to contact sampled persons. Unfortunately, such lists rarely exist in practice. Thus, survey researchers have devised methods for deriving sampling frames and for sampling from them. Box 5 lists several common types of sampling strategies used in health care surveys and identifies their pros and cons.

Sampling strategies can be quite complex. Most large Federal surveys use multiple stage sampling, where, for example, local areas, dwelling units, and finally individual families are enumerated and selected in successive stages. Such strategies can lead to considerable savings of survey administration costs, but require advanced statistical methods both for sample selection and data analysis. Even more modest sampling efforts may require complex strategies to reach the comparatively small population groups that are often of considerable interest in health access surveys. The performance of the health care safety net can best be judged by how well it serves low-income persons with chronic health conditions. If, for example, 1 in 20 individuals in a given community falls into such a population of special interest, then to interview 100 such individuals would require a sample of 2,000 persons selected randomly.

Box 5. Sampling Strategies for Health Surveys

Sampling Strategy Description Pros Cons
Random digit dialing

Frame: All listed and unlisted telephone numbers

Random selected telephone numbers within selected exchanges. Often involves screening to eliminate non-residential numbers; prescreened lists are available to survey firms

  • Inexpensive
  • Methods are well-developed and scientifically accepted
  • Enables limited geographic targeting based on telephone exchange
  • Applicable only to telephone surveys
  • Excludes households without telephones
  • Complicated by the growing role of wireless telephones
  • Generally precludes sending advance letters and other materials
  • Response rates generally lower than more personalized methods
Area probability samples Frame: Residents in a selected geographic areas

Random selection of geographic areas and households within the areas

  • Can increase efficiency of in-person interviewing
  • Does not exclude persons without telephones
  • Enables geographic targeting
  • Requires enumeration of areas and households
  • Generally limited to in-person interviewing
  • Difficult to reach homeless families and those doubling up in households
  • Expensive
List samples

Frame: Available lists

Random selection from list

  • Can be highly efficient for identifying members of target population
  • Inexpensive
  • Lists of target population are rarely available
  • Quality and completeness of lists are often in question
Intercept samples Frame: All persons in designated place

Approach individuals where they naturally congregate (e.g., emergency department waiting room)

  • Inexpensive
  • Can reach highly selected target populations
  • Non-scientific sample
  • Limited to short questionnaires
  • Requires permission of facility sponsor
  • Privacy may be lacking

To avoid collecting large amounts of survey data from populations that are of little interest, survey researchers often use screening techniques. Screening involves asking a few questions at the outset of a survey, and excluding cases that are not of interest. In safety net-related surveys, however, screening often involves asking about income, which is highly sensitive and can lead many study subjects to refuse to be interviewed.

Another technique for identifying populations of interest is targeting based on information available for other sources. In area probability sampling, for example, data from the U.S. Census can be used to identify low-income areas. As noted in Box 5, however, this technique is expensive. Random digit dialing (RDD) and list sample strategies may also permit limited geographic targeting, although telephone exchanges are generally not adequately geographically limited to enable efficient sampling based on income or local demographics. Survey sampling experts have developed sampling targeting and screening techniques that may be applicable for the efficient administration of local safety net assessment surveys.

Telephone Coverage Bias

Because of its relative economy and expediency, most local health-related surveys use random digit dialing (RDD) sampling strategies. Given that the users of access studies are often vitally interested in low-income families, many of whom do not have telephones, methods have been devised to reduce the bias that can result from excluding families without telephones. Some surveys supplement RDD sampling with in-person interviews drawn from small area probability samples of families without telephones. This method is costly and raises complexities that require advanced statistical expertise. Alternatively, researchers have begun to ask telephone survey respondents whether they recently lacked a home phone. Persons with a history of lacking a telephone have been found to be quite similar in their characteristics to those who lacked a phone on the day of the survey, thus data analysis techniques can be used to statistically adjust to minimize the telephone bias. These adjustments also require advanced statistical expertise.

If survey planners lack resources to engage statistical experts to help them address the telephone coverage bias problem, then some simple steps are possible to document the possible extent of bias from missing families without telephones. First, include a telephone history question in the survey. Second, when analyzing survey findings, examine the outcomes of interest by whether families lacked a phone recently. Such individuals might, in fact, experience greater access problems than other families. This method will also enable a rough estimate of how many families might have been missed because they lacked a telephone and how their health access might differ from that of households with telephones. Table 5 shows an example of how an analysis might be presented. This table suggests that somewhere in the neighborhood of 7 percent of the target population could not be interviewed during the month of the survey (this assumes that few families permanently lack a telephone) and that non-phone families are more likely to lack coverage and to experience worse access to care.

Selecting the Sampling Strategy

Often, cost is the major factor in selecting the right sampling strategy. However, other factors should be considered. The extent to which advanced, scientific sampling strategies are needed depends on the study goals. If, for example, the survey is intended to "sway" a skeptical audience that access problems are widespread, then careful, rigorous sampling is a must. Alternatively, if the purpose of the survey is to identify emerging issues and concerns for programming planning purposes, then samples that are less scientific (and less expensive) might be acceptable. For example, if clinic managers wish to find ways to improve services, then a brief survey of persons completing clinic visits might suffice. Such a sample would clearly represent only a selected group of clinic users; it might miss, for example, individuals in a hurry or those too ill to take time for an interview after a clinic visit. The key to deciding whether less scientific methods are acceptable is to clearly understand what sampling biases might result; that is, to understand who is likely to be missed and why.

Picking the Right Sample Size

Surveys reported in the popular press almost always report a range of survey error, often something like "plus or minus 3 percent." These reports are intended to give the reader a sense of how precise or accurate survey estimates are, and they reflect the number of persons interviewed, known as the sample size—one of many possible sources of imprecision in surveys. Survey precision also depends on sampling strategies and other factors. Identifying sample size needs requires the expertise of a professional statistical consultant (survey firms can generally provide sample size advice). However, if resources are not available for such a consultation, common sense can go a long way. Consider not just the total size of the sample, but think about which subgroups the survey is intended to represent. For example, if the African-American population is of specific interest in a survey project, but the community to be surveyed is only 10 percent African American, then a sample of 100 would yield only about 10 African Americans. Extrapolating from samples as small as 10 is not a good idea. Unfortunately, there is no hard rule about minimum group sizes. Some researchers believe that results for sample sizes of fewer than about 50 cases may be too small to be considered reliable. Whatever rule of thumb is used, however, thinking through subgroup analysis goals can help in determining sample size requirements.

Return to Contents

What to Ask

Another essential step in survey planning is identifying what to ask. Again, well-defined survey objectives are the starting point for deciding what to ask in a survey. It is tempting to stray from goals, especially when a committee is responsible for survey planning. It is advisable to write the agreed-upon survey goals on a large sheet of paper, tape it to the wall during planning sessions, and refer to it often!

Clearly, a survey questionnaire that asks about topics that are not essential to the survey goals can add to the cost of the survey, but an equally important reason to avoid questionnaire digressions is respondent burden. Survey respondents will tire quickly, and asking them more than is necessary will exact a cost in respondent refusal to participate and quitting the interview before it is completed. Also, if respondents do not understand why they are being asked certain questions, then data quality and respondent participation rates will suffer. So, stay focused!

Before selecting or writing survey questions, translate the survey goals into broad questionnaire domains, or subject areas. Table 3 outlines key domains for access to care surveys (e.g., coverage, access to care, and demographics). Then, within each domain, identify the data elements that are necessary to collect (Table 3). Finally, proceed with the task of identifying or writing specific survey questions. As discussed, it is advisable to draw questions from existing questionnaires whenever possible. The first part of this chapter identifies candidate sources of questions for most domains of interest for safety net-related surveys. Write new questions only as a last resort.

In selecting measures and survey questions, some general guidance may be helpful. The following sections highlight specific measurement issues in several important domains.

Measuring Insurance Coverage

Most surveys assessing the safety net will need to collect information about population health insurance coverage (and lack of coverage). As discussed earlier in this chapter, measuring health insurance coverage is more difficult than is commonly assumed. The complexity of health coverage in the United States contributes to the complexity of surveys about coverage. For example, it is not always clear that certain ways of paying for health care are truly insurance coverage. For example, U.S. military veterans are eligible for services through the Department of Veterans Affairs, but even VA system users may not consider this coverage. As well, some individuals may not immediately equate health maintenance organization (HMO) membership with health insurance. Thus, asking a simple survey question about whether a respondent has health insurance is likely to lead to misleading results. For that reason, survey experts believe that accurate measurement of whether and what type of health coverage an individual has is best achieved by asking about each possible source of coverage. Individuals who respond "no" to each source of coverage may then be considered uninsured.

Another aspect of health coverage that may confound survey responses is that most types of private insurance have one policyholder who may then cover one or more dependents. Typically, surveys ask about policyholders (i.e., the person in whose name the policy is held) and then ask who else within the family may be covered under the plan. It is important to note that some people may be covered by policyholders living outside the household, such as dependents of a divorced or separated spouse who has moved out. Most batteries of coverage questions in national surveys account for such possibilities.

When selecting coverage questions, the timeframe of coverage is another important consideration. The main national survey tracking coverage, the CPS, asks about coverage in the prior calendar year. In this survey, the "uninsured" are those who said they did not have coverage from any source in the prior calendar year. It is widely believed that the CPS coverage measure is not accurate and comes closer to measuring the number of uninsured persons at a single point in time (Swartz, 1986). Respondents may have difficulty recalling exactly when coverage began or ended, particularly over long recall periods. Thus, designers of most health access-related surveys use coverage at a single point in time (e.g., the day of the survey or the month before the interview) as their core measure. Table 3 shows the coverage concepts that national surveys use to measure coverage.

Although it is preferable to measure coverage at a point in time, local survey designers should be aware that the CPS is the most often cited source of national and State-level coverage estimates. This fact should not necessarily, however, lead local survey developers to adopt the CPS coverage questions. Indeed, for a myriad of methodological reasons (many of which are discussed in this chapter), even adopting the exact wording of CPS coverage questions does not guarantee that local survey results will match their statistics. Whatever measurement strategy is adopted, local survey sponsors should be prepared to explain why their estimates of coverage might vary from such official statistics. Good preparation for addressing questions about why local survey coverage estimates vary from those reported by others is important; after all, a public debate about survey measurement problems upon release of local survey findings can be a terrible distraction from discussions of more important substantive access issues.

Measuring Access to Care

There are many possible ways to conceptualize and construct survey measures of access to care (Millman, 1993). Box 6 lists some common measures used in population surveys and identifies their potential strengths and weaknesses.

Because each measure has strengths and weaknesses, the best access studies use a combination of measures to glean as complete a picture as possible. Studies that rely only on single measures can be misleading. For example, one of the most commonly used indicators of access, having a usual source of care (USC) other than an emergency room, is considered important. Having a USC can enable access and good continuity of care, which is especially important for people with complex conditions such as diabetes. However, some individuals may meet their access needs without a USC. Studies have shown, for example, that most people without a USC do not want one and that not having a USC may result from low health care needs (Hayward et al., 1991; Kuder and Levitz, 1985). In addition, the growing prevalence of managed care plans that require enrollment with a primary care gatekeeper may lead some individuals to report having a USC because the "system" expects them to have one (Cantor, 1998). Over the past 20 or so years, the proportion of persons reporting not having a USC has been roughly stable, but the percent with a USC among groups that we might expect to have the most access problems (e.g., uninsured and Hispanics) has risen (Zuvekas and Weinick, 1999).

The extent to which survey questions are more or less subjective is an important consideration in selecting access measures. Some questions that are easiest to ask and most intuitively clear are also the most subjective. For example, many surveys ask whether respondents had delayed or forgone care that they felt they needed. Clearly, this measure depends on perceptions of ones own needs, which may vary greatly by respondent expectations, knowledge, and beliefs. Measures that are highly subjective may be subject to conditioning bias in questionnaires. Conditioning bias occurs when the answers to questions are affected by their placement in the questionnaire. For example, respondents may be less likely to say that they experienced delays in getting needed care if these questions are placed after a battery of questions about care utilization compared to placement after health status questions. Also, reports of delayed and forgone care tend to be higher when questions are asked separately about specific types of care (e.g., general medical, dental, and mental health) than when asked as one general question (Berk, Schur, and Cantor, 1995).

Box 6. Common Access Measures Used in Population Surveys

Measure Strengths Weaknesses
Usual source of care
  • Considered an important enabler of access and important for continuity of care
  • Fairly easy and efficient to ask
  • May reflect individual preferences as much as access to care
  • May not reflect barriers to using care, although followup questions can be added (e.g., satisfaction with current usual source)
Delayed or forgone care and perceived access
  • Easy and efficient to ask
  • Findings intuitive and easily understood
  • Highly subjective. For example, respondents with low expectations may fail to report real access problems
  • Does not explain causes of barriers, although followup questions can be added. However, such questions generally elicit only general, often cost-related, responses (e.g., "could not afford to go")
Specific barriers and experiences with care (e.g., appointment waiting time, travel time)
  • Can identify specific problems
  • Fairly easy and efficient to ask
  • Reflects only care barriers that are evident to the respondent
  • Satisfaction-type questions are subjective and mediated by respondent expectations, making comparisons among groups hard to interpret
Doctor visit in prior year
  • Easy and efficient to ask
  • Many medical experts recommend that everyone should visit a doctor at least annually
  • Objective measure
  • May be subject to recall bias; i.e., it may be hard to recall events over a long period
  • Young, healthy people may not need an annual doctor visit
Preventive services use
  • Objective measure of expert-recommended care
  • Fairly easy and efficient to ask
  • Recommendations for use of some preventive tests lack medical consensus
  • May not reflect access for illness care
  • May be subject to recall bias

It is not critical that local survey developers learn all of the nuances of commonly used access measures, as even experts disagree about the interpretation of many of these measures. However, it is advisable to use a variety of access questions, including some that are comparatively specific and objective (e.g., visit to a doctor in past year, use of recommended preventive care) and some that are broader but more subjective (e.g., delayed and forgone care). Taken together, such measures can be used to paint a statistical portrait of the characteristics of people experiencing access difficulties. In addition, if local survey results are to be compared with other surveys, it is advisable to adopt consistent question wording and ordering to avoid finding differences that might result from methodological idiosyncrasies.

Measuring Health Status

Another important measurement domain for safety net assessment surveys is health status. As with all aspects of survey development, it is vital to review survey objectives before embarking on selecting health measures. Survey-based health status measures can serve many possible purposes: they can be helpful in understanding the level of medical care needs in a population, and, in some circumstances, they can be useful for assessing outcomes of medical care and other kinds of interventions. The fact that health status measures reflect both needs and outcomes complicates their use and interpretation. For example, if people who use a hospital outpatient department have lower health status than those who use a community health center, are they getting worse care? Obtaining an answer to this question is difficult, and the types of surveys discussed in this chapter are not adequate tools for studying such questions. Thus, care must be taken in interpreting health status questions. In this example, it is certainly fair to say that outpatient department users reported more health needs than health center users on the day of the survey. However, it is not possible to say whether the hospital patients' needs are the result of poor care or the result of the fact that sicker patients may be more likely to seek care at the hospital.

Other important considerations when selecting health status measures in surveys related to the safety net include:

  • Clarifying what dimensions of health need to be measured.
  • Identifying measures that are appropriate for the survey population of interest.
  • Considering the practicality of survey administration and data analysis.

"Health" is a complex and multidimensional concept; it can imply "wholeness" and a general feeling of well-being or it can imply an absence of symptoms, illness, or pathology. It can also extend to concepts of functional ability or absence of disability. Several domains of health are often considered distinct: mental health, dental health, and physical health, for example. When considering issues of access to the health care safety net, it is vital to consider which of these domains are of interest. Safety net users commonly experience mental health and dental problems as well as other medical problems.

Another important consideration in selecting health status measures is the population of interest. The focus here is on general population surveys (as opposed to surveys of disease-specific or other kinds of patient populations). Health measures can be defined positively, i.e., the degree to which one's health is good, or negatively, i.e., the degree of problems experienced. In general populations, few individuals will report experiencing any given health problem. Thus, surveys that rely on negative health reports must ask about a large number of health conditions or the results will reveal nothing about the health of most survey respondents. For this reason, using positive reports about health can be of greatest value (Ware et al., 1981).

After clarifying the dimensions of health of interest and the characteristics of the population to be surveyed, survey designers must consider the practicality of applying health status measures. Given the many possible health conditions and dimensions of health that might be included in a survey, respondent burden is a consideration. Some health status survey batteries are quite long and could result in respondent fatigue. These same measures can be complex to analyze, requiring the construction of composite indices. The temptation to pick and choose from among individual items on heath scales should be avoided, because many are designed as a package and omitting elements can compromise their validity. On the other hand, some measures of general health require little interviewing time and have a high level of validity.

Perhaps the most commonly used general health measure could not be simpler:

  • Would you consider your health to be excellent, very good, good, fair, or poor?

Remarkably, this well-studied measure has been shown to be highly predictive of death rates and other "objective" indicators of health, future functional ability, and health services use (e.g., Idler and Benyamini, 1997; Idler and Kasl, 1995). For other dimensions of health, namely functional status, settling on a single measure to meet all needs is difficult. Some disability measures focus on work (or school) days lost due to illness, but these will not identify individuals who are retired or those with disabilities who have adjusted their lives (e.g., using assistive devices) so that they are able to work or attend school. The most commonly used measures of functional status among older persons ask about a list of activities of daily living (e.g., bathing and shopping for personal items).

With these considerations in mind, survey developers should refer to the surveys reviewed at the outset of this chapter to select measures. Those inclined to go further can refer to the rich published literature about health measures (Ware, 1995 or National Center for Health Statistics, 2003).

Locale-Specific Questions

Clearly, local surveys afford opportunities to ask the questions of greatest interest to local decisionmakers. Among the potentially most useful advantages of surveys tailored to local needs is the ability to ask about specific local institutions and circumstances. National and State surveys often attempt to identify utilization patterns using generic terms, asking respondents whether they used "clinics" or "hospital outpatient departments," but respondents do not always use these terms as the survey designers intend. For example, when asking about the type of usual sources of care, respondents in Minnesota may be more likely to respond "clinic" than respondents in New York. This result may not be the result of a greater role for community clinics in Minnesota, home of the Mayo Clinic, but rather the types of institutions to which respondents apply the term.

Local surveys can avoid the pitfall of using generic terminology by asking about specific institutions. In-person surveys of literate populations can use cards listing local health facilities or offices to help respondents identify their source of care. Likewise, over the telephone, interviewers can read a list of providers (if there are not too many) or ask the name of providers used and record them verbatim to be looked up later.

In any case, this information can be linked to sources describing these locations (e.g., a provider survey) when the data are analyzed. Comparative information about specific providers can be a powerful tool for local advocates to motivate service changes among the providers. (Care must be taken, however, not to extrapolate from very small sample sizes, as discussed above).

Collecting other kinds of locale-specific information also can be helpful. For instance, knowing local demographics can help shape questions about ethnic background, and knowledge of local transportation systems can help shape questions about barriers to getting to care sites. A little creativity in adapting questionnaires to local circumstances can go a long way in enhancing the value of surveys.

Classification Variables

Local surveys also will need a range of variables to describe the kinds of outcomes already discussed. These classification variables will generally include questions covering the items listed in Box 7. Like other kinds of survey questions, local survey designers should not reinvent the wheel, as there is a wealth of examples of survey items available for creating classification variables. Like other kinds of survey content, there are often complexities in asking about those characteristics that are not immediately obvious, so relying on proven questions is vital. Tables 1 and 2 identify potential sources of survey questions for two of the more complex domains of classification variables, employment and income. All health surveys also include questions about the other domains listed in Box 7.

Box 7. Classification Variables

  • Demographics (age or year of birth, sex, race, and ethnicity)
  • Immigration status
  • Family structure and relationships
  • Employment status
  • Income and wealth

Return to Contents
Proceed to Next Section

 

The information on this page is archived and provided for reference purposes only.

 

AHRQ Advancing Excellence in Health Care