Mortality Measurement: Mortality Risk Adjustment Methodology for University Health System Consortium
By Steven J. Meurer, Ph.D., M.B.A., M.H.S
The University Health System Consortium (UHC) risk adjusts length of stay (LOS), costs, and mortality in the Clinical Data Base (CDB). Additionally, complications are risk adjusted in the CDB using a complication profiler. The risk adjustment process for LOS, costs and mortality involves four steps:
- Assignment of a Severity of Illness (SOI) and Risk of Mortality (ROM) level to each cas.
- Selection of a patient population to serve as the basis of the model to provide norms.
- Use of regression techniques to predict probability of mortality and LOS and costs based on the normative patient population.
- Assignment of an expected probability of mortality, LOS and costs to every patient in the CDB.
1. Assignment of Severity of Illness and Risk of Mortality Levels
Patient characteristics impact resource utilization and clinical outcome greatly. To ensure equitable inter-hospital comparison of outcomes, it is necessary to adjust for differences in patient characteristics. A number of patient classification schemes have been developed to address the issue of patient severity. While each of these schemes defines severity differently, most are similar in one respect: they use specific combinations patient demographics and principal and secondary diagnoses and procedures to define different levels of severity and complexity of treatment.
The All Patient Refined Diagnosis Group (APR-DRG) Grouper, developed by 3M Health Information Systems, is used by UHC to assign a level of illness severity and a risk of mortality level for each patient in the CDB. The APR-DRGs were developed as an expansion of the DRG system to address patient severity of illness and risk of mortality as well as resource intensity. The development process for the APR-DRGs involved an iterative process of formulating clinical hypotheses and then testing the hypotheses with historical data. Separate clinical models for severity of illness and risk of mortality were developed for each of the base APR-DRGs and then tested with historical data to review the clinical hypotheses. In the APR-DRG system a patient is assigned three distinct descriptors:
- The base APR-DRG (e.g., APR-DRG 044 Intracranial Hemorrhage or APR-DRG 194 Heart Failure).
- The severity of illness (SOI) subclass.
- The risk of mortality (ROM) subclass.
The underlying clinical principle of the APR-DRGs is that the SOI or ROM subclass of a patient is largely dependent on the patient's underlying problem and that patients with high SOI or ROM are usually characterized by multiple serious illnesses. The determination of the SOI and ROM is disease specific, i.e., the significance attributed to comorbid conditions is dependent on the underlying problem. High SOI and ROM are primarily determined by the interaction of multiple illnesses involving multiple organ systems which result in patients who are difficult to treat and tend to have poor outcomes.
The four SOI and ROM subclasses are:
Although the subclasses are numbered sequentially, the numeric values represent categories and not scores. It is not meaningful to average the numeric values of the SOI or ROM subclasses across a group of patients to compute an average severity score. The SOI or ROM subclasses can be used as categories within which category-specific rates for an outcome are calculated to represent an expected value for that SOI or ROM subclass.
2. Identification of Patient Population for Model Generation
The identification of a patient population for model generation is a very important step of building risk adjustment models and should be linked with its purpose. "Broadly construed, risk adjustment attempts to account for all factors other than the health care intervention itself or the process of care that may explain variation in patient outcomes. Four factors may account for observed differences in outcomes: differences in how well available data sources represent reality; differences in significant risk factors among patients; random variation; or differences in the effectives of the health services provided or the process of care". (Iezzoni, 1997 page 200) A poorly identified patient population will not allow a clinician accurately determine where differences in outcomes occur.
The three components of UHC's identification of a patient population for model generation include using the Medicare Severity Diagnosis Related Groups (MSDRG) classification system, using only patients that entered one of our academic medical center members, and cleaning the data. UHC believes that choosing a model population must take into account the homogeneity or a group as well as appropriate sample size. For example, a model could be built with patient's that are similar in demographics and comorbid conditions, but the sample would be so low that a robust model could not result. At the other end of the spectrum, all patients in a very large dataset could be considered the model population, but overfitting would result as few similarities would result. Therefore, UHC has chosen to use our academic medical center patient's from the last two years in a base MSDRG as our model population. We only use patients from our academic medical centers in order to be the most relevant to our owners. In total, these 85 hospitals add approximately 2.7 million inpatient discharges to the database each year. UHC does not collect hospice patients. Finally, cases flagged as "bad data" and transfers out to another acute care hospital are excluded from the model population.
3. Regression Models
Logistic regression models are constructed for the binary outcome variable in-hospital mortality, and multiple regression models are developed for the continuous outcome variables of LOS and cost. Separate models are built for base MS-DRG. In 2007, UHC developed 337 models each for LOS, costs and mortality. Strict exclusion rules are applied before building the regression models. For example, only base MS-DRGs with more than 100 cases and an incidence rate greater than 1% are modeled for mortality. Due to the potential for serious estimation problems related to a sparse contingency table, another layer of exclusion criteria is also imposed, i.e., observed incidence counts (rate multiplied by number of cases) must be greater than 50. Independent variables included in all the regression models are:
- ROM subclass for mortality, and SOI subclass for LOS and costs.
- Patient Age.
- Patient Sex.
- Admit source = transfer from another acute care hospital.
- Admit source = transfer from skilled nursing facility.
- Admit source = long term care facility.
- Low socioeconomic status (SES) (based on Medicaid, Self-Pay, Charity as primary payer).
- Admit status = emergency.
- Patient Race.
- Comorbid conditions defined by AHRQ, based on research by Elixhauser, et al, including:
- Congestive heart failure
- Valvular disease
- Pulmonary circulation disorders
- Peripheral vascular disorders
- Hypertension (complicated and uncomplicated)
- Other neurological disorders
- Renal failure
- Liver disease
- Peptic ulcer disease excluding bleeding
- Metastatic cancer
- Solid tumor without metastasis
- Rheumatoid arthritis/collagen vascular diseases
- Weight loss
- Fluid and electrolyte disorders
- Blood loss anemia
- Deficiency anemias
- Alcohol abuse
- Drug abuse
Each year, clinician input and literature review is used to add additional variables for specific patient populations. For example, palliative care identified by ICD-9-CM diagnosis code V66.7 is included in models for conditions such as heart failure, stroke, AIDS, and malignancies.
Additional diagnoses and/or procedures specific to certain patient populations such as neonatology, oncology, transplants, pediatrics, obstetrics, spinal surgery, ventilator support, cardiac surgery, cardiology, neurosciences, and other high impact DRGs and Base MS-DRGs are included in the models for those patients. UHC updates the models each year to include the latest 2 years worth of academic medical centers. It is at this time that recommendations from a member committee on risk adjustment are included in the risk adjustment process.
A standard cross validation statistical method is used to ensure the stability of the parameter estimates. Data are randomly split into training and validation samples. Key model diagnostics include the c-index and the Hosmer-Lemeshow test. A c-index ≥0.70 is generally accepted as indicating good discrimination by a model. No mortality models with c-index values <0.70 are accepted for the CDB. The final regression models are then constructed using the full data set. When a mortality model for base MS-DRG fails the validation tests, each case in the DRG or Base MS-DRG is assigned the observed mortality value stratified first by the ROM subclass from the APR-DRG grouper and then by whether or not the patient was transferred from another acute care hospital. This occurs mainly for one of two reasons: the incidence of the outcome variable is too low, or the patient population is small. Base MS-DRGs with very low observed mortality (≤1% or <50 deaths) are not modeled for mortality. For those base MS-DRGs, expected mortality is based on observed mortality stratified first by ROM subclass within the DRG or Base MS-DRG and then by transfer status. For mortality models, goodness of fit tests (e.g., Hosmer-Lemeshow tests) are run to determine how well the model predicts the outcome. Only models that "pass" the goodness of fit tests are used to estimate expected probability of mortality.
Potential causes of biased differences between observed and expected values derived from regression models include the following:
- Omitted Risk Factor(s)—If factors that are significantly related to outcome are not included in the risk adjustment model, the model can underestimate the expected values for hospitals having a disproportionate share of patients with the missing risk factor(s).
- Biased Reporting of Risk Factor(s)—If a hospital consistently under or over reports risk factor(s) with influential effects on the model estimates, then its expected values will be systematically biased. Since severity of illness estimation, risk of mortality estimation, and comorbidity counts depend on the accuracy of ICD-9-CM coding, review of medical records for accuracy is very important.
- Biased Statistical Model—If the statistical model is not well calibrated and tested for systematic over or under estimation, then the model may systematically over or under estimate expected values. The literature has shown that this type of bias is not important for analyses at the individual patient level. However, when aggregating to the hospital level, model-related bias may become influential, since institutional differences were not fully accounted for by the model.
To minimize the effects of these factors, hospitals should carefully evaluate coding consistency, completeness and accuracy, including the coding of UB-04 fields like admission source and admission status, as well as ICD-9-CM diagnosis and procedure coding.
4. Application of Models to Entire Database
The output from a stable model includes a coefficient for each of the variables found to be statistically significant predictors of the outcome, i.e., mortality in the population that was modeled. Using the coefficients for each of the statistically significant predictor variables and a flag indicating whether the variable was present in the patient, UHC then calculates a predicted value for each patient in the CDB. All patients in the database are assigned predicted values for mortality, LOS and costs. The example below illustrates the calculation of expected probability of mortality for a stroke patient (DRG 14).
Model Group: # 14 - Spec CV dis exc TIA (DRG 14)
Model Diagnostics: Calculation: Chi-sq = 20.411 Validation: Chi-sq = 38.383, F = 1.881, p = 0.1803 Final: Max VIF = 2.789, Hosmer-Lemeshow = 48.622, p = p <0.001, df = 10, C = 0.822
Mean Observed = 0.1334, Mean Expected = 0.1334
Cases = 21,394
Model Method = Logistic Regression
Model Results (Significant Predictors)
|Coeff||Explanatory Variable||Coeff||Explanatory Variable|
|3.292||ROM = 4 (Extreme)||-0.423 CC||Fluid & Electr Disorders|
|2.856||Palliative Flag||-0.490||CC Congestive Heart Failure|
|1.830||ROM = 3 (Major)||-0.609||CC Peripheral Vasc Disease|
|0.500||Female, Age ≥85||-0.649||CC Psychoses|
|0.489||Male, Age ≥85||-0.663||CC Deficiency Anemias|
|0.473||Female, 80 ≤Age <85||-0.772||CC Diabetes w/ CCs|
|0.448||CC Renal Failure||-0.807||CC Pulm Circulation Disease|
|0.417||Male, 80 ≤Age <85||-0.845||CC Depression|
|0.414||Admission Source = Emergency||-1.177||CC Pept Ulcer Dis X Bleed|
|0.402||Admit Src = Transf From Acute||-1.304||CC Acq Immune Def Synd|
|-0.190||CC Diabetes w/o CCs||-1.600||Male, 1 ≤Age <18|
|-0.320||CC Hypothyroidism||-1.652||CC Weight Loss|
|-0.392 CC Valvular Disease|
Patient Y Characteristics
Age 54 \
Transferred from another hospital.
Emergency admit status.
APR-DRG ROM = Extreme.
Principal Dx = 430 (subarachnoid hemorrhage.)
CC Drug abuse.
CC Coronary artery disease.
CC Asymptomatic HIV status.
CC History of DVT/embolism with long-term anticoagulant use.
For each of the significant explanatory variables from the model determine whether that variable was present for this patient. If a variable is present it receives a value of 1, if not present its value is 0. For each of the significant predictor variables, multiply the variable's coefficient from the model times the patient value for that variable:
|ROM = 4 (Extreme)||1||3.292||3.292|
|ROM = 3 (Major)||0||1.830||0|
|Female, Age ≥85||0||0.500||0|
|Male, Age ≥85||0||0.489||0|
|CC Renal failure||0||0.448||0|
|Adm stat emergency||1||0.414||0.414|
|Adm src trns frm acute||1||0.402||0.402|
|CC Diabetes w/o CCs||0||-0.190||0|
|CC Valvular disease||0||-0.392||0|
|CC Fluid & electro dis||0||-0.423||0|
|CC Periph vasc dis||0||-0.609||0|
|CC Defic anemias||0||-0.663||0|
|CC Diabetes w/ CCs||0||-0.772||0|
|CC Pulm Circ Dis||0||-0.807||0|
|CC Pept ulc dis exc bld||0||-1.177||0|
|CC Acq imm def synd||0||-1.304||0|
|Male, 1 <= Age <18||0||-1.600||0|
|Male, 1 <= Age <18||0||-1.600||0|
|CC Weight loss||0||-1.652||0|
Formula for calculating expected mortality for a case
where n = # of significant variables or risk factors; vari = 1 or 0 for variable i present in this patient or not ; coeffi = coefficient for variable i
exp is the exponential function or exp(x) = ex (e.g., exp(0) = 1, exp(1) = 2.718, etc.)
Using this formula Patient Y's expected probability of mortality = 0.4960.
To ensure that the models are well calibrated for each of the fiscal years to which they are applied, intercept values and coefficients for each of the variables found to be statistically significant during model generation are re-calculated for each fiscal year. The intercept values and coefficients from the model results for the most recently completed fiscal year are applied to the next fiscal year's data until that fiscal year is completed. At that time the models are re-calibrated for the just completed fiscal year. This process is then repeated for each future fiscal year.
Iezzoni LI, Foley SM, Heeren T, Daley J, Duncan CC, Fisher ES, Hughes J. A method for screening the quality of hospital care using administrative data: preliminary validation results. QRB (1992 Oct) 361-71.
Iezzoni LI, Daley J, Heeren T, Foley SM, Fisher ES, Duncan C, Hughes JS, Coffman GA. Identifying complications of care using administrative data. Medical Care (1994 Jul) 32(7):700-15.
Iezzoni LI, Daley J, Heeren T, Foley SM, Hughes JS, Fisher ES, Duncan CC, Coffman GA. Using administrative data to screen hospitals for high complication rates. Inquiry(1994 Spring) 31(1):40-55.
Iezzoni LI, Heeren T, Foley SM, Daley J, Hughes J, Coffman GA. Chronic conditions and risk of in-hospital death. Health Serv Res (1994 Oct) 29(4):435-60.
Iezzoni LI, Shwartz M, Ash AS, Hughes JS, Daley J, Mackiernan YD. Severity measurement methods and judging hospital death rates for pneumonia. Medical Care (1996) 34(1):11-28.
Iezzoni LI, Shwartz M, Ash AS, Mackiernan YD. Does severity explain differences in hospital length of stay for pneumonia patients?. J Health Serv Res Policy (1996 April) 1(2):65-76.
Iezzoni LI. Assessing quality using administrative data. Ann Intern Med (1997 Oct 15) 127(8 Pt 2):666-74.
Iezzoni LI (Ed). Risk adjustment for measuring healthcare outcomes (2nd edition). Chicago, IL: Health Administration Press; 1997.
Weingart SN, Iezzoni LI, Davis RB, Palmer RH, Cahalane M, Hamel MB, Mukamal K, Phillips RS, Davies DT, Banks NJ, Davies DT Jr. Use of administrative data to find substandard care: validation of the complications screening program. Medical Care (2000 Aug) 38(8):796-806.
Hosmer DW and Lemeshow S. Applied logistic regression (2nd edition). New York: John Wiley & Sons, Inc.; 2000.
Avrill RF, Goldfield N, Hughes JS, Bonazelli J, McCullough EC, Steinbeck BA, Mullin R, Tang AM, Muldoon J, Turner L, Gay J. All Patient Refined Diagnosis Related Groups (APR-DRGs) Version 20.0: Methodology Overview. Wallingford, CT: 3M Health Information Systems; 2003.
Elixhauser A, Steiner C, Harris DR, Coffey RM. Comorbidity measures for use with administrative data. Medical Care (1998) 36(1):3-5.
Current as of February 2009