This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: https://info.ahrq.gov. Let us know the nature of the problem, the Web address of what you want, and your contact information.
Please go to www.ahrq.gov for current information.
Chapter 2. Methods
This chapter briefly summarizes key issues related to the methods used to generate the first NHDR. Critical components of the analysis plan that are common to all succeeding chapters are highlighted. Methods that are unique to particular issues or measures are discussed in their appropriate chapter. Detailed methods pertaining to each survey database analyzed are available in an appendix.
In general, the methodological approach of the NHDR is to examine a broad array of measures from a variety of data sources, standardize data and comparisons as much as possible, and apply uniform and rigorous thresholds for identifying significant differences. This approach allows the NHDR to present findings that are typically more comprehensive, detailed, and accurate than similar information available to health care organizations from other sources.
A key guiding principle of the NHDR is consistency with Federal guidelines and publications. Paramount among these documents is Healthy People 2010.1 Like Healthy People 2010, the NHDR addresses a broad range of issues and populations and uses data from a large number of sources. Like Healthy People 2010, it seeks to standardize definitions across data sources, but allows for differences when standardization is not achievable. As much as possible, methodological consistency with Healthy People 2010 is sought. However, this cannot always be achieved. When these methods deviate from Healthy People 2010, this is indicated.
Many groups participated in the development of the methods used in this report. One essential group is the Interagency Work Group for the National Healthcare Disparities Report. This group includes representatives from:
- Administration for Children and Families
- Administration on Aging
- Assistant Secretary for Health
- Assistant Secretary for Legislation
- Assistant Secretary for Planning and Evaluation
- Assistant Secretary for Public Affairs
- Centers for Disease Control and Prevention (CDC)
- CDC-National Center for Health Statistics
- Centers for Medicare & Medicaid Services
- Food and Drug Administration
- Health Resources and Services Administration
- Indian Health Service
- National Institutes of Health
- Office of Civil Rights
- Office of Minority Health
- Substance Abuse and Mental Health Services Administration
Congress directed AHRQ to produce a report on "prevailing disparities in health care delivery as they relate to racial factors and socioeconomic factors in priority populations."2 Key methodological issues include:
- Defining racial, ethnic and socioeconomic categories
- Selecting measures of disparity in health care and appropriate data sources
- Specifying analyses of disparities
- Presenting findings
Return to Contents
Definition of Racial, Ethnic, and Socioeconomic Categories
Racial and Ethnic Categories
Different data sources collect information on race and ethnicity in different ways. The NHDR team considered two options for categorizing race and ethnicity:
Option 1: Use racial and ethnic categories that are compliant with OMB standards for reporting Federal statistics.3 These standards consider race and Hispanic origin to be two separate and distinct concepts. Racial categories are: white, black, Asian, Native Hawaiian or Other Pacific Islander (NHOPI), American Indian or Alaska Native (AIAN), and more than one race. All racial categories can include persons of Hispanic and non-Hispanic origin. Ethnic categories are: Hispanic or Latino, non-Hispanic white, and non-Hispanic black. Hispanics can be of any race.
Option 2: Use racial categories that could be supported by all data sources. This lowest common denominator approach would entail collapsing racial categories to accommodate the least-refined data sources.
With input from the NHDR Interagency Work Group and others, the NHDR team decided upon the former. However, because Federal data collection was not required to be compliant with OMB standards until 2003 and much data available for this report were collected in 1999 and 2000, many findings in this report use different racial and ethnic categories. Whenever data sources rely on racial and ethnic categories other than those identified by OMB, the report includes the definitions and labels that were used during its data collection process and notes differences from OMB specifications. While this creates different race and ethnicity categories for measures from different data sources, it retains more information than collapsing categories to meet a lower standard and is consistent with categories used by Healthy People 2010.
There is no consensus about the best way to measure socioeconomic position as it relates to health care.4 Socioeconomic position is typically measured using income or education. For example, Healthy People 2010 categorizes persons based on their family income level relative to poverty thresholds and on their educational attainment.
Whenever feasible, the NHDR includes information on both family income and education. Education was categorized using Healthy People 2010 categories:
- Less than high school
- High school graduate
- Any college education
Family income level relative to poverty thresholds was also categorized using Healthy People 2010 categories, except that middle/high was divided. Specifically, the NHDR classifies income as follows:
- "Poor" means below 100% of the Federal poverty level (FPL)
- "Near poor" means 100-199% of the FPL
- "Middle income" means 200-399% of the FPL
- "High income" means 400% of the FPL or more.
Dividing the middle/high category was necessary because much of the disparities research demonstrates differences between middle and high income persons.
When income or education was not available, other proxies for socioeconomic position were sought. These include insurance status and median income of a person's ZIP Code of residence. Individual income and the median income of the area where the individual lives are highly correlated.5 However, individual income and area income may influence health care differently. While individual income is primarily a measure of individual resources that could be applied to facilitate health care, area income may encompass other effects such as the availability of health care in the area. Hence, area income is used to measure socioeconomic position only when no other measure is available.
Return to Contents
Selection of Measures and Data Sources
Selection of Measures
Many measures of disparity in health care exist. A major task of the NHDR team and the NHDR Interagency Work Group was to identify and select measures of disparity for the first report. While consistency of measures from year to year is highly desirable, the measures selected for inclusion in the first NHDR represent a small subset of currently available measures and are expected to evolve as the field of health care measurement itself evolves.
The selection of measures of disparity in health care to include in the first NHDR was guided by two key principles, used whenever possible:
- Measures developed through consensus processes, whereby experts convene and deliberate with the goal of producing high quality measures
- Measures consistent with Federal guidelines and publications
Because Healthy People 2010 includes many objectives of relevance to the measurement of disparities in health care and was produced by the collaborative work of many Federal, State, and private organizations, its measures are used whenever applicable. When consensus measures are unavailable, NHDR includes measures that are commonly used by researchers in the field.
Measures followed two separate pathways for inclusion in this report.
- First, because this report and the National Healthcare Quality Report (NHQR) are companion documents, NHDR adopted the quality of care measure set for the NHQR in its entirety. In this first edition of the reports, the quality of care measures are identical in both the NHDR and the NHQR except for several measures for which analysis by race/ethnicity and socioeconomic position was not possible.
- Second, disparities can exist in many aspects of health care delivery other than quality of care. To compile measures that relate to these other areas where health care disparities exist (i.e., access to care, use of care, and cost of care), AHRQ published a call for measures in the Federal Register on June 5, 2002, and engaged the Institute of Medicine to convene experts to hear public testimony, commission papers, and provide guidance on this report.
The full measure set is available at http://www.ahrq.gov/qual/nhdr02/prenhdr.htm as well as in the Appendix.
Input From Disparities Data Experts
AHRQ formed the NHDR Data Group, an internal committee comprised of AHRQ staff with expertise in health care disparities and data analysis. These staff members helped develop preliminary access, utilization, and cost measures relevant to studying disparities and identify data sources for these measures. The measures were reviewed and augmented by the NHDR Interagency Work Group.
After a preliminary measure set was published on the AHRQ Web site, public input was again solicited. This process yielded the set of measures used in this report. They are organized into two major topic areas:
- Access to Health Care
- Quality of Health Care.
Access to health care covers issues related to whether persons can get health care or experience barriers to care. Measures relate to entry into the health care system, structural barriers within the system, the ability of providers to address patient needs and health care utilization. Quality of health care covers issues related to whether persons getting health care receive the services that they need. Measures relate to the effectiveness, safety, timeliness, and patient centeredness of services received.
Selection of Data Sources
After identifying the major topic areas, the specific measures to be included and the data source for each measure were identified. The criteria used for selecting data sources for each of the identified measures are as follows:
- The data sources have to provide data by race, ethnicity, and/or socioeconomic status.
- Nationally representative data are preferred in order to allow for national estimates. However, when such data are not available, the NHDR team relies on non-Federal and/or regional data sources.
- Because of the small size of many populations of interest, the NHDR team favors databases with large sample sizes to increase the likelihood of reliable information for the population subgroups.
- If a measure is identical to one included in Healthy People 2010, the NHDR uses the same database.
- To enable tracking of trends over time, data collected periodically are emphasized over one-time efforts.
- To maximize consistency between both reports, the NHDR and the NHQR use the same data sources for shared measures. However, there are several exceptions. For several quality of care measures, NHQR uses databases that do not include reliable information for selected racial or ethnic groups. In these instances, NHDR seeks alternative sources with more reliable subpopulation data.
To address gaps in Federal data collection related to cultural competency and health care information, the NHDR team used 2001 data from the Commonwealth Fund Health Care Quality Survey. The response rate for this survey was not as high as the others included in this report. Gaps in available HIV data were filled by data collected by the HIV Research Network. To allow more detailed examinations of Hispanic and Asian subgroups and of American Indians and Alaska Natives, the NHDR used data from the California Health Interview Survey. This survey is the largest State health survey in the U.S. and collects information about health status and health care in six languages.
In total, the NHDR integrates data from over 20 different data sources. Federal data sets used in the NHDR include major data holdings maintained by the Agency for Healthcare Research and Quality, Centers for Disease Control and Prevention, Centers for Medicare & Medicaid Services, National Centers for Health Statistics, National Institutes of Health, and Substance Abuse and Mental Health Services Administration. Details of individual data collections are available in the Appendix. A brief listing of databases used in the NHDR is included below:
Surveys collected from samples of civilian, noninstitutionalized populations:
- AHRQ, Medical Expenditure Panel Survey (MEPS), 1998-2000
- California Health Interview Survey (CHIS), 2001
- CMS, Medicare Current Beneficiary Survey (MCBS), 1999
- The Commonwealth Fund, Health Care Quality Survey, 2001
- NCHS, National Health and Nutrition Examination Survey (NHANES), 1999-2000
- NCHS, National Health Interview Survey (NHIS), 1998 and 2000
- NCHS, National Immunization Survey (NIS), 2001
- SAMHSA, National Household Survey of Drug Abuse (NHSDA), 2000.
Data collected from samples of health care facilities:
- CMS, End-Stage Renal Disease Clinical Performance Measurement Program, 2001
- CMS, Nursing Home Resident Profile Table, 2001
- NCHS, National Ambulatory Medical Care Survey (NAMCS), 1999-2000
- NCHS, National Home and Hospice Care Survey (NHHCS), 2000
- NCHS, National Hospital Ambulatory Medical Care Survey-Outpatient Department (NHAMCS-OPD), 1999-2000
- NCHS, National Hospital Ambulatory Medical Care Survey-Emergency Department (NHAMCS-ED), 1999-2000
- NCHS, National Hospital Discharge Survey (NHDS), 1998-2000
- NCHS's National Nursing Home Survey (NNHS), 1999
- NIH, United States Renal Data System (USRDS), 2000
- SAMHSA, Client/Patient Survey Sample (CPSS), 1997.
Data extracted from administrative data systems of health care organizations:
- AHRQ, Healthcare Cost and Utilization Project State Inpatient Databases 16-State databasei (HCUP SID), 2000
- Medicare claim data from CMS
- HIV Research Network data (HIVRN), 2000.
Data extracted from medical records of health care organizations:
- CMS, Quality Indicators program, 1998-1999.
Population-based data collections:
- CDC, HIV/AIDS Surveillance System, 2000
- CDC, TB Surveillance System, 1998-1999
- NCHS, National Vital Statistics System (NVSS), 2000
- NIH, Surveillance, Epidemiology, and End Results (SEER) program.
Years of data included in this report range from 1997 to 2001. These represent the most recent years of complete data available for the report. To ease reading, findings are discussed in the present tense but reflect data from these years.
As with data selection, measure specification was guided by consistency with Healthy People 2010, the NHQR, and other Federal publications whenever applicable. In addition, when a specific measure was related to a Healthy People 2010 measure, NHDR sought comparable specification.
For example, "proportion of persons with health insurance" is a Healthy People 2010 measure, while "proportion of persons with public health insurance" is not. However, because the two measures are related, they are specified in a comparable fashion: restricted to persons under 65 and with age adjustment.
Return to Contents
Examination of Disparities
Several decisions guided the comparisons that were made for this report. Criteria for comparisons relate to the selection of reference groups for comparisons and the choice of a measure of disparity. The NHDR team sought consistency with existing Federal guidelines and publications so that comparisons would be as easily understandable as possible.
Disparities are typically defined relative to one or more reference points. Within DHHS, data experts are working to specify a reference point standard for disparities for Healthy People 2010, but consensus has not yet emerged. Therefore, the NHDR team considered three basic options for reference points. First, groups could be examined relative to the total population. Second, groups could be examined relative to the "best performing" group. Third, groups could be examined relative to the largest fixed group.
The NHDR team adopted the third option, reasoning that it would be the easiest to understand since reference groups would not change from measure to measure as in comparisons with the "best performing" group. In addition, because each reference group is numerically the largest, standard errors tend to be the smallest for this group. Moreover, unlike comparisons with the total population, groups are independent.
The largest fixed groups are whites for racial comparisons, non-Hispanic whites for ethnic comparisons, 400% or more of the FPL for income comparisons, and any college education for education comparisons.
This choice of comparison groups is not meant to suggest that whites or persons with high income or college education are superior in any way or that disparities are an issue for racial and ethnic minorities or less affluent persons only. In fact, whites and persons with high income or college education are not the "best performing" group in many instances.
Disparities can be measured as absolute differences or as percentage differences from a reference point. Disparities can be shown for individual groups compared with a reference group, or summary statistics can be calculated that incorporate disparities for individual groups into a single value for the population as a whole. As mentioned above, groups are working in DHHS to specify a disparities measurement standard as well as to create summary measures of disparity for the population as a whole for Healthy People 2010. In the absence of consensus, it was decided to measure disparities in relative terms as percentage differences compared with the reference group. This option appears to be the easiest to understand. Data for individual groups are available in the appendix.
Return to Contents
A number of decisions guided the specific analyses conducted for this report. These decisions relate to suppressing data that are unreliable and identifying the most significant disparities. In general, the NHDR team established high thresholds for data suppression and determination of significance in order to concentrate on the highest quality data and the most significant disparities.
Different data systems apply different criteria to suppress data deemed unreliable, ranging from no suppression of data to complicated algorithms. In an effort to standardize the quality of data across the many data systems providing information for this report, the NHDR team decided to impose two minimum data suppression criteria across all databases:
- Estimates based on sample size less than 30
- Estimates with relative standard error (standard error divided by parameter estimate) greater than 30% when appropriate
Hence, almost all data in this report are based on cell sizes of at least 30 with a relative standard error of 30% or less, when appropriate. Any exceptions in the report are noted. Specific data suppression criteria for each database are available in the Appendix.
These criteria were applied in addition to standard data suppression criteria maintained by individual data systems. When data systems had more rigorous suppression criteria than those adopted by the NHDR team, the more stringent criteria were maintained. For example, the National Health Interview Survey suppresses data with cell sizes less than 50 and the Medical Expenditure Panel Survey suppresses data with cell sizes less than 100. NHIS and MEPS data presented in this report adhere to these more rigorous standards.
A general exception to these data suppression criteria is data that encompass population counts, such as data from vital statistics. Such data are not subject to error related to sampling and typically have their own suppression criteria. For example, mortality statistics based on fewer than 20 deaths are typically suppressed; data presented in this report adhere to this criterion.
Determination of Differences as Significant
Different databases used in this report vary greatly in their size and scope. In some large databases, such as the HCUP SID 16-State database, even very small differences are often statistically significant. Other databases, such as the National Vital Statistics System, encompass population counts, making significance testing inappropriate. Criteria other than statistical significance were therefore needed to help focus on the most important disparities.
Hence, two criteria were specified for determining significance. First, differences between each priority population and its reference group must be statistically significant with p<0.05. Second, relative differences of at least 10% from the comparison group are tagged as significant.
A related issue is whether disparities should be examined in terms of favorable outcomes, adverse outcomes or both. For example, Healthy People 2010 specifies many measures in terms of favorable outcomes (e.g., percent of persons with health insurance, percent of persons with an ongoing source of care, etc.). Because achievement of these favorable outcomes is relatively high in the total population, relative differences tend to be minimized compared with examining adverse outcomes (e.g., percent of persons without health insurance, percent of persons without an ongoing source of care, etc.). Because the report includes measures that range from under 5% to over 95% of the general population, the NHDR team examined all relative proportions in terms of both favorable outcomes and adverse outcomes. Relative differences of at least 10% in one or the other comparison were used to label a difference as significant.
For example, the percent of persons under 65 with health insurance, a favorable outcome, is 85% among whites and 80% among blacks (NHIS, 2000). Calculating the difference between these two groups (blacks minus whites) relative to the comparison group (whites), (80%-85%)/85%.6%, this difference does not reach the 10% criterion for a significant difference. However, if we convert the measure to its adverse outcome, percent of persons under 65 without health insurance, the white rate is 15% and the black rate is 20%. Now, the relative difference (20%-15%)/15%.33%, does reach the 10% criterion for a significant difference. Therefore, this report considers the black-white difference in health insurance to be significant.
In summary, this report considers differences significant if:
- The difference is statistically significant at the alpha=0.05 level, two-tailed test and
- The relative difference is at least 10% different from the reference group when framed positively as a favorable outcome or negatively as an adverse outcome.
For population-based data for which significance testing would be inappropriate, only the second criterion needs to be met in order to be labeled a significant difference. However, it is important to note that the absence of a significance label for a particular comparison does not mean that disparities do not exist. Differences may be of a magnitude of less than 10% or may not achieve statistical significance due to small sample sizes.
Because of the many measures and multiple comparisons that are an integral part of this report, it was recognized that focusing on specific differences would be problematic. Explicit adjustment for multiple comparisons was not performed. Instead, analyses focused on disparities that cut across multiple groups or across multiple related measures. Typically, disparities that involved all or most minority groups or all or most lower socioeconomic groups are presented. Greater weight is also given to differences that involve only one or two groups but that involve multiple related measures.
Because identifying disparities involves some subjectivity, the review process sought to be inclusive and iterative. Summary data tables containing all comparisons and identifying those that met criteria for significance were reviewed by members of the NHDR Interagency Work Group, AHRQ staff, and stewards of the data set employed.
Return to Contents
Presentation of Findings
The NHDR illustrates findings related to the general population in three ways. First, the appendix contains summary tables and detailed tables for all measures. Summary tables of disparities in the general population across all measures are presented to allow readers to see all data supporting this report, both areas with significant disparities and areas without. For the access and quality chapters, the summary tables identify when a priority population performs worse than the comparison group as well as when a priority population performs better than the comparison group. Detailed tables allow readers to see and evaluate all the primary data for themselves.
Second, summary tables included in the report body present information for measures deemed to be most useful for assessing disparities. Third, report text focuses on key findings. Key findings are illustrated with simple bar charts showing differences in representative measures. Whenever possible, Healthy People 2010 measures are selected to be presented graphically; if none is available for that topic, the bar graph that appears in the text depicts a measure that affects larger numbers of people.
AHRQ priority populations are listed in the Introduction. Given the sample size constraints, as well as the large numbers of measures, comparisons, and demographic groups, the NHDR does not present the same level of detail for each priority population as it does for the general population. Instead, the report underscores how specific priority populations are unique by highlighting disparities for a specific priority population when such disparities differ from the general population. (Go to Chapter 5.)
Return to Contents
Challenges and Limitations
Analyzing the data in this report required weighing conflicting needs and interests to present a balanced view of those areas of health care in which disparities do and do not exist. Providing an overview of a broad range of measures precludes in-depth examination of each one.
For each measure, the report includes data on racial and ethnic disparities across each priority population, stratified by socioeconomic status (as recommended by the IOM), but does not include multivariate analyses or measures at the intersection of multiple priority populations (e.g., racial disparities among low-income women). In addition, the report emphasizes data at the national level rather than at the State or local levels. Ultimately, then, the first report favors a broader scope of measures over more detailed analysis of each measure.
Few Data on Subpopulations
Related constraints were posed by the availability of data for subpopulations. While important differences in health care exist within some of the populations examined, such as among Hispanic and Asian subpopulations from different countries of origin, many data sets do not collect this level of detailed data on race and ethnicity. Even among those that do, small sample sizes generally preclude such analysis.
Ultimately, the report relies on the racial and ethnic categories specified by the Office of Management and Budget for the collection of Federal data. As noted earlier in this chapter, racial categories include white alone, black alone, American Indian and Alaska Native alone, Asian alone, Native Hawaiian and Other Pacific Islander alone, and more than one race; ethnic categories include Hispanic or Latino (of any race), not Hispanic or Latino white race alone, and not Hispanic or Latino black alone.3 The main data included in the report based on this classification are supplemented by data from the California Health Interview Survey, one of the few large survey efforts with adequate samples to address subpopulation issues.
No Universal Definition for Socioeconomic Disparities
While OMB guidance is available to help specify racial disparities, comparable standards do not exist to help specify socioeconomic disparities. Researchers use a variety of measures of socioeconomic position including income, poverty, education, occupation, wealth, class, and social capital; consensus does not exist about which measure is best for examining disparities in health care. Thus, as previously mentioned, in the absence of specific guidance, the NHDR focuses on family income relative to Federal poverty thresholds and education as commonly used and available measures of socioeconomic position and sought to include both dimensions when feasible.
Finally, the capacity to measure the existence of racial, ethnic and socioeconomic disparities in health care far exceeds the current state of knowledge explaining why such disparities exist and how to reduce them. Given the breadth of the Congressional mandate to provide a national overview of disparities in health care, the NHDR focuses on documenting existing disparities. The first report will provide a baseline from which to track future trends in health care disparities.
Return to Contents
1. U.S. Department of Health and Human Services. Healthy People 2010. Washington, D.C.: 2000.
2. Public Law 106-129.
3. OMB, Provisional Guidance on the Implementation of the 1997 Standards for Federal Data on Race and Ethnicity, 12/15/00, available at http://www.whitehouse.gov/omb/inforeg/re_guidance2000update.pdf [PDF Help].
4. Gornick, IOM, Guidance for the National Healthcare Disparities Report, Chapter 2, Measuring the Effects of Socioeconomic Status on Health Care, Academy of Sciences, 2002.
5. Krieger N, Williams DR, Moss NE. Measuring social class in US public health research: Concepts, methodologies, and guidelines. Annu Rev Public Health. 1997; 18:341-378.
Return to Contents
Proceed to Chapter 3: Quality of Health Care