Skip Navigation U.S. Department of Health and Human Services
Agency for Healthcare Research Quality
Archive print banner

Research Briefs

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to for current information.

Engels, E.A., Schmid, C.H., Terrin, N., and others (2000). "Heterogeneity and statistical significance in meta-analysis: An empirical study of 125 meta-analyses." (AHRQ grants HS08532, HS10064, NRSA training grant T32 HS00060). Statistics in Medicine 19, pp. 1707-1728.

Meta-analysis is used to synthesize results from randomized controlled trials in clinical medicine. However, an important issue is how to incorporate heterogeneity (variation among the results of individual trials beyond that expected from chance) into summary estimates of treatment effect. Another consideration is which metric to use to measure treatment effect. For trials with binary outcomes, there are several possible metrics, including the odds ratio (a relative measure) and risk difference (an absolute measure). This study of 125 meta-analyses found that for most meta-analyses, summary odds ratios and risk differences agreed in statistical significance, leading to similar conclusions about whether treatments affected outcome. Heterogeneity was common regardless of whether treatment effects were measured by odds ratios or risk differences. However, risk differences usually displayed more heterogeneity than odds ratios.

Lawthers, A.G., McCarthy, E.P., Davis, R.B., and others. "Identification of in-hospital complications from claims data: Is it valid?" pp. 785-795; McCarthy, E.P., Iezzoni, L.I., Davis, R.B., and others. "Does clinical evidence support ICD-9-CM diagnosis coding of complications?" pp. 868-876; and Weingart, S.N., Iezzoni, L.I., Davis, RB., and others, "Use of administrative data to find substandard care: Validation of the complications screening program." pp. 796-806. (AHRQ grant HS09099). August 2000 Medical Care 38(8).

The first study examined the validity of the Complications Screening Program (CSP), which uses claims data to identify hospital complications. The researchers wanted to find out if ICD-9-CM codes used to identify complications were coded completely and accurately and whether the CSP algorithm successfully separated conditions present on admission from those occurring in the hospital. The CSP screens for 28 potential complications, for example, postoperative pneumonia, using specific ICD-9-CM secondary diagnosis or procedure codes referred to as "trigger codes." Eighty-nine percent of the surgical cases and 84 percent of the medical cases had their CSP trigger codes corroborated by re-review of the medical record. For 13 percent of the surgical cases and 58 percent of the medical cases, the condition represented by the code was judged to be present on admission rather than occurring in-hospital.

The second study examined whether medical records contained clinical evidence supporting the ICD-9-CM discharge diagnosis and procedure codes used by the CSP to identify complications. Overall, 30 percent of medical and 19 percent of surgical patients lacked any documented evidence in the medical record, even physicians' notes. Rates of confirmatory clinical evidence varied widely across the complication screens. Some complications, such as postoperative heart attack, had most cases confirmed by explicit clinical criteria, while other complications had fewer than 60 percent of cases confirmed using clinical evidence. These findings raise serious questions about the clinical validity of CSP codes, which are increasingly being used to evaluate hospital performance and determine payment.

The third study used administrative data to validate whether the CSP identified complications of care and potential quality problems. The researchers stratified acute care hospitals by observed-to-expected complication rates and randomly selected hospitals within each State. They randomly selected cases flagged with one of 17 surgical complications and 6 medical complications. Physicians confirmed flagged complications in 68.4 percent of surgical and 27.2 percent of medical cases. They identified potential quality problems in 29.5 percent of flagged surgical and 15.7 percent of flagged medical cases but in only 2.1 percent of surgical and medical controls. The authors conclude that for some types of complications, screening administrative data may offer an efficient approach for identifying potentially problematic cases for physician review.

Seid, M., Varni, J.W., and Kurtin, P.S. (2000, August). "Measuring quality of care for vulnerable children: Challenges and conceptualization of a pediatric outcome measure of quality." (AHRQ grant HS10317). American Journal of Medical Quality 15(4), pp. 182-188.

This article addresses conceptual and practical issues in the assessment of pediatric health care quality, outlines a conceptual model for measuring quality, and describes ongoing research to validate an outcome measure of health care quality for vulnerable children. The authors point out that pediatric quality measurement is distinct from that for adults, due to development, dependence, differential epidemiology, demographic factors, and differences between the child and adult health service systems. They assert that a noncategorical approach to quality measurement, rather than one based on illness status or specific condition, is necessary to adequately measure quality for the majority of children, both healthy and ill. One promising noncategorical measure of pediatric health care quality is health-related quality of life (HRQOL). The Pediatric Quality of Life Inventory (PedsQL), a brief, practical, reliable, valid, generic pediatric HRQOL measure, is a suitable candidate measure, which the authors describe.

Return to Contents

Current as of September 2000
AHRQ Publication No. 00-0054

The information on this page is archived and provided for reference purposes only.


AHRQ Advancing Excellence in Health Care