Mortality Measurement: Slide Presentation (Text Version)
Slide Presentation for Mortality Measurement Meeting
Slide Presentation for Mortality Measurement Meeting
On November 3, 2008, Dave Foster, Ph.D., M.D., made this presentation at the 2008 Annual Conference. Select to access the PowerPoint® presentation (712 KB).
Slide presentations can be accessed using a Free PowerPoint® Viewer.
Background image showing a graph on a computer screen overlain with three photos of health care professionals
Institute for Healthcare Improvement (IHI) Mortality Measurement Meeting
David Foster, Ph.D., M.P.H.
Center For Healthcare Improvement
Thomson Reuters Healthcare
Each slide has Thomson Reuters Healthcare logo at the bottom.
- Thomson Reuters (TR) Projected Inpatient Data Base (PIDB):
- Combines data from both public and proprietary state data as well as individual and group hospital contracts.
- The construction of the PIDB involves the application of sophisticated data screens to ensure quality.
- Contains more than 20 million all-payer discharges throughout the U S from over 2,700 acute care hospitals (about half of the actual discharges that occur in the U.S. annually).
- Statistically projected using stratified sampling weight information to the entire U S population of acute care inpatient discharges.
- The PIDB has been used for many peer-reviewed publications.
- Comprised of administrative data.
Data Source (continued)
- Selected peer-reviewed articles published from PIDB data:
- Young J K, Foster D A, Heller S T. Cardiac revascularization in specialty and general hospitals.N Engl J Med 2005 Jun 30;352(26):2754-6
- Belay E D, Holman R C, Maddox R A, Foster D A, Schonberger L B. Kawasaki Syndrome Hospitalizations and Associated Costs in the United States. Public Health Reports 2003 Sep-Oct;118(5):464-9
- Foster D A, Heller S T, Young J K. Increasing Prevalence of Resistant Streptococcus among Hospital Inpatients in the United States. N Engl J Med 2001 344:1329-31 (correspondence)
- Young J K, Foster D A. Use of Cardiovascular Procedures after Acute Myocardial Infarction in Patients with Mental Disorders. JAMA 2000 283(24):3198; (correspondence) discussion 3198-9
- Sullivan K M, Delay E D, Durbin R E, Foster D A, Nordenberg D F. Epidemiology of Reye Syndrome, United States, 1991-1994: Comparison of C D C surveillance and hospital admissions data. Neuroepidemiology 2000 19(6):338-44
TR Risk-Adjusted Mortality Model (RAMI)
- Comprised of four standard logistic regression models:
- Less than 65 years of age, surgical
- Less than 65 years of age, medical
- 65 or more years of age, surgical
- 65 or more years of age, medical
- ICD-9-CM diagnosis and procedure codes that are considered intervening events, such as hospital-acquired complications, are excluded.
- A post-modeling adjustment based on AHRQ Clinical Classification Software (CCS) categories created from principal diagnosis is used to reduce the compression that typically results from regression models.
- Produces an expected probability of death for each patient.
TR Risk-Adjusted Mortality Model (RAMI)
- Patient-level risk factors:
- Age, sex, admission source, admission type
- Principal diagnosis, all other diagnoses codes, all procedure codes (ICD-9-CM) through use of risk-tables:
- Principal diagnosis
- Secondary diagnosis with highest risk
- Procedure code with highest risk
- Interaction between principal and secondary with highest risk
- Interaction between principal and procedure with highest risk
- Hospital-level adjustment factors (optional):
- Bed size category
- Teaching status
- Urban/rural community setting
- Census division
RAMI Facility Exclusions
- Long-term care facilities (typical Medicare discharge length of stay greater than 25 days)
- Cancer specialty hospitals
- Psychiatric, Substance Abuse, and Rehabilitation specialty hospitals
- Federally owned or controlled facilities
- Hospitals that are missing identified characteristics or have fewer than 6 beds
RAMI Patient Exclusions
- Invalid or incomplete data
- Inconsistent age, sex, diagnosis or procedure code interactions
- Encounter for palliative care
- DRG Not Surgical Or Medical
- DRG 468 Extensive Operating Room (OR) Procedure Unrelated To Principal Diagnosis
- DRG 477 Non-extensive OR Procedure Unrelated To Principal Diagnosis
- Other (Appendix C in RAMI white paper)
RAMI Example Of Risk Table
|DX code||PDX lt 65||SDX lt 65||PDX ge 65||SDX ge 65|
Model Performance Metrics
- Sensitivity measures the percent of patients correctly classified among those that experience the outcome.
- Specificity refers to the percent of patients correctly classified among those that did not experience the outcome.
- Percent correct describes the percentage of patients whose predicted outcome matches their actual experience, regardless of whether or not they experienced the outcome.
- C-Statistic: the area under a receiver operating characteristic (ROC) curve (maximum area equals 1.0).
ICD-9-CM Exclusions as Intervening Events
- Diagnosis code examples (from a total of 83):
- 2513 Post surgical hypoinsulinemia
- 3240 Intracranial abscess
- 3241Intraspinal abscess
- 3249 Intracranial and intraspinal abscess of unspecified site
- 38330 Postmastoidectomy complication, unspecified
- 41511 Iatrogenic pulmonary embolism and infarction
- 45821 Hypotension of hemodialysis
- Procedure code examples (from a total of 14):
- 0123 Reopening of craniotomy site
- 0302 Reopening of laminectomy site
- 0475 Revision of previous repair of cranial and peripheral nerves
- 0602 Reopening of wound of thyroid field
- 1152 Repair of postoperative wound dehiscence of cornea
- 1266 Postoperative revision of scleral fistulization procedure
Model Performance Results
|Medical, age ≥ 65||4.43%||0.915||85.4||76.8||89.6|
|Medical, age < 65||0.97%||0.978||94.5||90.8||94.9|
|Surgical, age ≥ 65||2.51%||0.963||91.7||87.7||92.7|
|Surgical, age <65||0.76%||0.986||95.7||94.7||95.8|
RAMI versus Disease Staging (DS) Mortality Results (ROC Curve)
ROC graph of RAMI and Disease Staging for In-Hospital Death
The x axis shows specificity on a scale of 0 to 1. The y axis shows sensitivity on a scale of 0 to 1. Three curves are shown, one with RAMI c-statistic of 0.963, one with c-statistic of 0.919, and one with c-statistic of 0.962. At specificity from 0 to 0.1, all three curves rise sharply to sensitivity between 0.75 and 0.9. Approach specificity of 0.2, sensitivity is between 0.9 and 0.95. Beyond that, the curves flatten as sensitivity reaches 1.
RAMI versus D S Mortality Results (continued)
Graph of RAMI and Disease-Staging Z-Score by Service Line (Service lines are sorted by the value of z-score, respectively)
The x axis shows Service Line from 1 to 27. The y axis shows z-score on a scale of minus 40 to 40. Two lines are shown, RAMI and DS. The z-scores for the RAMI service line range from minus 10 to approximately positive 13. The z-scores for DS service line range from about minus 22 to about positive 28.
RAMI versus DS Mortality Results (continued)
Spearman Correlation between Observed and Expected mortality—Patient Level
|Correlation||RAMI-Expected vs. Observed||DS-Expected vs. Observed||DS-Expected vs. Observed|
|Spearman Coefficients||0.34419 (p<0.0001)||0.24710 (p<0.0001)||0.56867 (p<0.0001)|
|Correlation||RAMI-Expected vs. Observed||DS-Expected vs. Observed||DS-Expected vs. RAMI-Expected|
|Pearson Coefficients||0.99241 (p<0.0001)||0.93435 (p<0.0001)||0.93373 (p<0.0001)|
RAMI versus A P R-DRG Mortality Results
Bar graph of Risk-Adjusted Mortality Predictive Value: RAMI versus A P R-DRG Version 20
Three sets of bars are shown.
The first set of bars shows sensitivity; RAMI, 89.9, A P R-D R G v20, 91.8, A P R-DRG v20: RAMI exclusions, 91.0.
The second set of bars shows specificity; RAMI, 89.1, A P R-D R G v20, 84.4, A P R-DRG v20: RAMI exclusions, 84.2.
The third set of bars shows accuracy; RAMI, 89.1, A P R-DRG v20, 84.5, A P R-DRG v20: RAMI exclusions, 84.4.
RAMI Performance as Described by External Investigators
Hall B L, Hirbe M, Waterman B, Boslaugh S, Dunagan WC. Comparison of mortality risk adjustment using a clinical data algorithm (American College of Surgeons National Surgical Quality Improvement Program) and an administrative data algorithm (Solucient) at the case level within a single institution. J Am Coll Surg 2007;205:767-77.
Conclusions: Risk-adjusted mortality estimates were comparable using administrative or clinical data. Minor performance differences might still have implications. Because of the potential lower cost of using administrative data, this type of algorithm can be an efficient alternative and should continue to be investigated.
RAMI versus National Surgical Quality Improvement Program (NSQIP) Mortality Results
Graph with specificity on the x axis and sensitivity on the y axis. Two lines are shown, one for Solucient and one for NSQIP. The c for Solucient is 0.976. The c for NSQIP is 0.937. Both lines rise sharply between specificity 0.0 to 0.1 Solucient rises to 0.95 and then levels out close to 1 for specificity above 0.1. NSQIP rises to 0.7 and continues rising to 0.9 at specificity of 0.2, leveling out beyond that to about 1.0.
"The c-statistics given reveal that the discriminatory power of both models was impressive: Solucient c equals 0.976, NSQIP c equals 0.937. The 95% confidence interval for the difference between these estimates does not include zero (0.07, 0.01), indicating a statistically significant difference in the discriminatory power of the two models, favoring Solucient."
Source: Hall BL, et al, J Am Coll Surg 2007;205:767-77.
- The RAMI methodology demonstrates high predictive value in comparing actual deaths with expected deaths.
- RAMI compares favorably with other risk-adjustment methodologies in terms of predictive value.
- RAMI benefits from a large calibration database that enables comprehensive consideration of interactions between principal diagnosis, other diagnoses and procedures.
- RAMI post-modeling adjustment does appear to mitigate the effects of model compression in comparison with a similar methodology that did not address compression.