Scientific Evidence Base for Quality Measure Development

Developing Quality of Care Measures for People with Disabilities: Summary of Expert Meeting

The group discussed the scientific evidence base available to support development of quality measures for persons with disabilities. Without this evidence, as Dr. Turk noted, often quality measures (e.g., designated "best practices") are based primarily on the opinions of practitioners. As an overall assessment, participants agreed that the body of research to inform quality metric development for persons with disabilities is limited.

Exceptions do exist. For example, certain disabling conditions, such as spinal cord injury and multiple sclerosis, have been fairly well studied. However, studies of disabling health conditions primarily focus on explicitly medical issues rather than psychoecological factors—they do not address the full range of disability dimensions. Few comparative effectiveness or patient-centered outcome studies have looked, for example, at the effects of community-based interventions, environmental modifications, or alternative models of care delivery (e.g., patient-centered medical homes, specific care coordination interventions). Even less information is available for children with disabilities, especially the life course of children living with disabilities and how interventions might work at various points in early life. A comprehensive research agenda is required to address these gaps.

Including Persons with Disabilities in Research Studies

Historically persons with disabilities have been automatically excluded from the randomized controlled trials (RCTs) that examine the comparative effectiveness of different disease-specific interventions (e.g., cancer treatments). This means that if a person with a disability develops that disease (e.g., cancer), the evidence base from RCTs may not apply to them. Not having RCT-level evidence can hamper efforts to develop scientifically rigorous quality measures for persons with disabilities.

In recent years, the National Institutes of Health (NIH) and AHRQ have required investigators submitting grant applications to explicitly address the inclusion of persons by sex and race and ethnicity. Both NIH and AHRQ grant applicants must justify the exclusion of people by sex, race, and ethnicity. NIH applicants must also address the inclusion of children and justify their exclusion. Beyond women and racial and ethnic minorities, AHRQ requires grant applicants to consider including the following "priority populations": inner-city residents; rural residents; low income persons; children; elderly people; and those with special health care needs, including individuals with disabilities and those who need chronic care or end-of-life health care.

The meeting participants discussed adding persons with disabilities to the list of population subgroups whose exclusion from RCTs or other clinical studies must be explicitly justified by investigators submitting grant applications to NIH and AHRQ. This change in policy would raise important questions, such as what would be considered appropriate reasons to eliminate persons with disabilities from clinical trials. Any such reasons must not be openly discriminatory or reflect erroneous assumptions about persons with disabilities (e.g., ability of an individual to provide informed consent). Nonetheless, important human subjects protection issues might arise. Another important question involves how to handle data on a relatively small subset of study participants who do have disabilities (e.g., do investigators have sufficient statistical power to say something definitive about persons with disabilities). Addressing the implications of the diversity of disabilities presents yet another critical issue.

Research Question (RQ): What are the factors that need to be considered in requiring researchers to justify the elimination of persons with disabilities from clinical studies, including RCTs? How should the diversity of disabilities, small sample sizes, and legitimate efforts to protect human subjects be addressed to maximize inclusion of persons with disabilities in clinical studies?

One specific type of observational research is ripe for the inclusion of people with disabilities: studies using Medicare claims files or other Medicare data sets. Often researchers exclude Medicare beneficiaries under age 65 (i.e., those qualifying for Medicare because of disability; end-stage renal disease is another special category of excluded cases) without explicit justification. Given that persons with disabilities constitute approximately 15 percent of the Medicare population, excluding them means that research results will not pertain to a substantial fraction of Medicare beneficiaries. Furthermore, given the size of the Medicare population, this 15 percent subset is likely to represent sufficient numbers of cases to perform meaningful subgroup analyses for many research questions. Participants agreed that all research involving Medicare beneficiaries should include persons who qualify because of disability unless there are strong and compelling reasons to exclude them. In other words, all AHRQ- and NIHfunded research using Medicare-generated data should include persons with disabilities unless their exclusion is clearly justified.

RQ: All studies using Medicare data should include beneficiaries under age 65 eligible because of disability. What would be considered legitimate reasons to allow investigators to exclude this subgroup of Medicare beneficiaries?

Despite strong interest in including persons with disabilities in research studies, participants recognized some of the challenges as described above (e.g., sometimes small sample sizes, clinically legitimate concerns about protecting human subjects). The group therefore considered alternative approaches for gathering data about effectiveness of interventions for persons with disabilities. Dr. Weinrich suggested the possibility of adding disability information to data gathering for post-market drug and device surveillance by the Food and Drug Administration (FDA). In addition to routinely gathering basic demographic data on persons experiencing events related to drugs or devices, the FDA should add information about the person's functional status or disability. Over time, these routine data would snowball into sufficient sample sizes for meaningful statistical analyses.

RQ: Would adding information on persons' functional status or disability to routine data gathering by the FDA for post-marketing surveillance of drugs and devices be an appropriate way to track the effectiveness of these interventions for persons with disabilities? If this seems like a productive approach, what data gathering approach would maximize the completeness and accuracy of the information about disability? How would these post-marketing surveillance data be used exactly to identify health risks and benefits for persons with disabilities?

Relationship to General Quality Metrics

Common health conditions that can be profoundly disabling include some, such as diabetes and heart failure, have quality measures that generally are widely accepted and used. Most of these quality indicators reflect processes of care (e.g., measurement of Hb A1c levels, ophthalmologic examinations, prescriptions for certain medications). These quality indicators do not address considerations relating to disability.

Dr. Kirschner raised the concern about considering disability effects in efforts to develop quality measures that cut across broad populations. Her example involved osteoporosis, as follows:

Kirschner: Osteoporosis is a huge problem for persons with disabilities.... Medically, we know a fair amount about the treatment of osteoporosis... For most of my patients, though, osteoporosis relates to immobilization.... Do we treat their osteoporosis the same way we do for post-menopausal women? Probably not.

RQ: As quality metrics are designed that apply to large numbers of patients with common conditions, are there special considerations for persons with disabilities? How should any special concerns relating to disability be factored into the quality measure?

Research on Care Delivery System Changes

Considerable discussion focused on the promise of patient-centered medical homes and approaches to improve care coordination but also the lack of clear research evidence about their merits for persons with disabilities. Costs of various care delivery models also require exploration, considering as well the implications for the health care workforce. Dr. Andresen raised the possibility of altering reimbursement structures to provide incentives for care relating to wellness and prevention. She also suggested research about ways to engage patients in self-management activities and having them become "a partner in the outcomes of their care." Research concerning disease selfmanagement has investigated conditions like diabetes, which are disabling. However, disability effects have not necessarily been the major focus of these studies.

With the growing interest in pay-for-performance reimbursement schemes, questions arise about whether these programs could affect persons with disabilities. Dr. Stineman worried about whether quality metrics designed for pay-for-performance would apply to persons with disabilities, and if not, would clinicians either: (1) avoid persons with disabilities or (2) provide inappropriate care to them? As Dr. Stineman asked, "If we move towards pay-for-performance, what are the special protections that people with disabilities need in place to make sure that there are not perverse incentives" relating to their care? If, for example, persons with disabilities should be eliminated from certain denominators in calculating rates for performance indicators, what is to stop clinicians from gaming these numbers (e.g., excluding persons inappropriately from denominators in rate calculations)?

A specific care delivery issue with insufficient evidence involves the transition into adulthood for youth with disabilities. Dr. Kirschner gave some examples:

Kirschner: I've been working a lot with adults with spina bifida. It's a great example of where we really need to be ahead of the curve in thinking about prevention of complications. We've got a generation of people with spina bifida who started getting surgeries in the 1960s, 1970s. They had great pediatric centers of excellence that provided coordinated care with the urologists, orthopedists, primary care doctors, rehab doctors, and neurosurgeons. Once people turn 21, pediatric hospitals don't want them anymore. There is nothing in the adult care system set up to meet their needs, and we're seeing lots of problems—really dangerous problems.

Cystic fibrosis is a wonderful example. In cystic fibrosis, centers of excellence really make a huge difference. The life expectancy of people with cystic fibrosis has changed dramatically from about 18 to around 32 years old. The centers of excellence have shown that [health care professionals] who do this day in and day out, who understand cystic fibrosis, can really have a dramatic impact on the quality of care.

Participants discussed the implications for creating centers of excellence of the small numbers of cases within given locales. Small numbers could be particularly problematic for rural areas. Telemedicine and other technologies to facilitate communication between remote and centralized centers of excellence might hold promise in these situations.

RQ: What models of care delivery address the bio-psycho-ecological needs of persons with disabilities and improve their outcomes? How much do these models cost, and what are the implications for the health care professional workforce?

RQ: What are the implications of different reimbursement structures for persons with disabilities? Are there payment models that effectively improve self-management activities among persons with disabilities? How might pay-for-performance programs affect persons with disabilities?

RQ: How can high quality care be delivered in remote locations with few persons with disabilities and staffed by a health care professional workforce without specific expertise? What is the role of telemedicine and other information technologies?

Methods Research

Participants mentioned several areas where additional methodological work is required. Developing strong outcome metrics related to care delivery is necessary. Examples include measures of health-related quality of life for persons with disabilities and outcome measures that could be tracked across the lifespan. Dr. Clark suggested looking at "gaps between what people want to do or need to do, like work, and whether or not they're able to do it. That [approach] might start to get at quality of life in a different way." Another area where more instrument design is needed is environmental assessments. According to Dr. Krahn, "We really have not moved very far in measures of environment. It might be... to look within the [environmental] context at factors that either facilitate or impede you" and design more rigorous, usable instruments to capture these factors.

Dr. Marge raised concerns relating to potential biases in peoples' self reports, especially of sensitive information (e.g., abuse by a caregiver). Obtaining self-reported information, including about satisfaction with care, from persons with intellectual disabilities also requires study. In particular, the role and potential biases of proxy respondents for persons with intellectual disabilities needs exploration.

Small sample sizes often present challenges to research on persons with specific disabilities. The small numbers problem may, in the end, be intractable, but additional methodological study should explore whether any new approaches might offer some benefits for studies involving small sample sizes. For example, the value of Bayesian analytic approaches should be examined in this context. Additional research on getting the most out of observational data (e.g., propensity scores have been a significant advance) could be helpful. Dr. Stineman mentioned research attempting to reduce selection bias using administrative data.

With the increasing availability of electronic health information, research should also explore the use of health information technology (health IT) and electronic medical records (EMR) in routine health care settings to capture disability information and support research concerning the care of persons with disabilities. Dr. Sandel is based at Kaiser Permanente, which has extensive health IT resources. She described the value of these data:

Sandel: I work with the large databases within Kaiser Permanente. We have 2,900 patients with a history of polio in our Northern California system. I haven't done any research on them, but if you get a large enough database, you can do some pretty interesting work.... We're [also using the data to look] at stroke mortality.... We have about 29,000 patients in the database over a period of time.... If you have a large enough database... you'll be able to do some pretty significant work, although it's not controlled randomized trials.

Dr. Himmelstein noted that the current emphasis on setting standards for meaningful use of health IT and EMRs offers opportunities for considering how to include information about disability.

RQ: What outcomes are critical to track in disability-related quality measurement research? What new outcome indicators must be developed, and how should data on these outcomes be collected? Can we create better measures to capture environmental factors that either facilitate or impede participation?

RQ: Are there new analytical approaches to using small samples of cases, observational data, or administrative data that could advance studies relating to quality of care for persons with disabilities?

RQ: How can the growing availability of health IT and EMRs be tapped to support disability-related quality measurement research? What should go into health IT systems and EMRs to gather credible and analyzable information relating to disability for quality measurement purposes?


Dr. Iezzoni summarized the discussion as follows:

  • With few exceptions, little scientific evidence is available to inform development of quality indicators for persons with disabilities.
  • Hence, not surprisingly, relatively few quality measures exist specifically addressing disability-related issues.
  • Those examples that do exist focus primarily on biomedical aspects of underlying disabling conditions rather than on functioning, wellness, quality of life, and the broad range of environmental concerns.

Examples where some quality measures do exist include those developed under the ACOVE (Assessing Care of Vulnerable Elders) initiative and for conditions like spinal cord injury, cystic fibrosis, and stroke. However, even among existing measures, few encompass the various domains required by a multidimensional definition of disability (e.g., ICF's definition, Table 2).

Page last reviewed September 2010
Internet Citation: Scientific Evidence Base for Quality Measure Development: Developing Quality of Care Measures for People with Disabilities: Summary of Expert Meeting. September 2010. Agency for Healthcare Research and Quality, Rockville, MD.