Skip Navigation Archive: U.S. Department of Health and Human Services U.S. Department of Health and Human Services
Archive: Agency for Healthcare Research Quality
Archival print banner

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to for current information.

Role of Partnerships: Second Annual Meeting of Child Health Services R

Advanced Models and Methods, Part I

Second annual meeting held to explore the state of the science in children's health services research.

Advanced Models and Methods, Part I


Christopher Forrest, M.D., M.P.H.
Assistant Professor, Johns Hopkins School of Hygiene and Public Health

Lawrence C. Kleinman, M.D., M.H.A.
Chief of Health Studies, Lehigh Valley Hospital

Allison Evans, B.A., M.B.A.
Center for Mental Health Services Research, University of California, Berkeley

James M. Perrin, M.D.
Director, Mass General Hospital Center for Child and Adolescent Health Policy

Scott Shipman, M.D.
Robert Wood Johnson Foundation Clinical Scholars Program, Johns Hopkins University


Introducing New Methods
Disparities in Specialty Referrals in Medicaid
Managed Mental Health Care in Colorado
Patterns of Care, Expenses and Hospital and Emergency Room Use by Children with Chronic Conditions

Introducing New Methods

Christopher Forrest, M.D. introduced the purpose of the breakout sessions on Models and Methods as raising awareness of emerging methodological issues that impact the findings and results of health services research. He provided background on the three key tools that were subsequently illustrated in case studies presented by the other panelists. These included:

  • Multi-level models for analyzing nested observations.
  • One vs. two-part models for analyzing utilization and expenditure data.
  • Risk adjustment for provider studies.

Multi-level models involve cases where lower-level observations are nested within higher level contexts, for example, patients (lower-level observation) of the same provider (higher level context). The problem raised by this situation is that patients seeing the same provider are more likely to be similar to one another than to those patients seeing other providers, i.e., they are not independent, but are dependent because of the same "context". One way to address this is through analysis of intra-class correlation (ICC), e.g., a measure of the degree to which patients share a common experience. However, with ICC the variance estimates may be off; the presence of ICC underestimates standard errors. Increasingly it is challenging to present research without addressing this issue. An alternative approach is a 2-level contextual analysis, including the Generalized Estimating Equation (GEE).

One challenging aspect of analyzing population use and cost data is that there is typically some portion (often about 20 percent) of a population that never uses services and there is a small percentage of people that account for the majority of users. Data are usually non-normally distributed. One approach to this type of data is the use of a two-part model: first a logistic regression that models probability of any use, and next an OLS (ordinary least squares) of users only. The two parts provide insights into the "causes" of differences: probability and/or amount of use.

Analysis of provider patterns of care as they relate to cost and use data may vary by different levels of comorbidity or measures of patient severity, and are therefore critical to address in reviewing system delivery design.

Return to Contents

Disparities in Specialty Referrals in Medicaid

While there has been a significant amount of research on access to primary care, there has been less research on access, use and quality of specialty care for underprivileged children. Scott Shipman described a study that compared Medicaid and commercially insured patients with respect to the following questions:

  • How does the referral decision differ?
  • Do primary care pediatricians coordinate specialty referrals differently (schedule the appointment, transfer information to/from the specialist)?
  • Are outcomes of referrals different (pediatrician satisfaction with specialty care, quality of specialists' communication)?

In the study, children referred by 142 primary care pediatricians in a network of physicians across 36 States was reviewed. (The network was the American Academy of Pediatrics' Network for Pediatric Research in Office Settings, or PROS). There were 1,807 referrals out of 58,771 encounters.

The conclusions of the study were that patients with Medicaid were referred more commonly than other patients. Further, coordination of referrals differed in that there was increased coordination by primary care providers for Medicaid patients, and decreased amount and quality of feedback from specialists for Medicaid patients. Pediatricians were less satisfied with specialty care for their Medicaid patients, and this diminished satisfaction was associated with less and poorer specialist feedback for their Medicaid patients.

One methodological issue considered in the study was the possible lack of independence of patient-level observations due to clustering within physician groups. The Generalized Estimating Equation (GEE), outlined earlier by Dr. Forrest, was used to address this; it was found that in this particular study, using the GEE to account for clustering of patient visits within providers did not substantively alter statistical conclusions.

Another methodological issue that was addressed was generalizability with respect to the physicians in the study and the patients in their practices. Comparison of the PROS pediatricians with a national sample showed similarities in age, gender, percent with a fellowship, years in practice, hours in clinical care, and diagnosis-specific referral rates. There were differences between the two physician populations; the PROS physicians spent more time in research, were less likely to be hospital-based, had fewer managed care patients, and served fewer underprivileged children.

PROS physicians were also grouped by the proportion of Medicaid patients in their practice. The "high-Medicaid" group was more likely to refer patients, schedule a specialty appointment, and communicate patient information with a specialist.

Dr. Shipman concluded by pointing out additional research opportunities to pursue related to these studies, including: analysis of the referral process from the perspectives of the patient/parent and the specialist; analysis of the referral process for uninsured children for whom there are likely to be greater barriers; and implementation and evaluation of efforts to improve referral coordination and primary care-specialist communication.

Return to Contents

Managed Mental Health Care in Colorado

The policy interest of the study described by Allison Evans-Cuellar was to better understand the incentives for providers to shift care among managed care services, or outside of managed services. The study considered Medicaid eligible youth that used mental health services under Colorado's managed mental health contracts. Claims and encounter data were reviewed for 45,975 youth, including 4505 who were in the juvenile justice system and 9007 who were in the child welfare system. The goal of the study was to look at the change in utilization resulting from managed care relative to a "baseline" change in a fee-for-service system. The expectation was that there is greater potential for shifting where there are multiple systems, and where there are high-risk clients, such as those in juvenile justice or child welfare.

Three models of care were analyzed, including two systems operating under Colorado's managed mental health carve-out:

  • Non-profit community mental health centers under contract to the State.
  • Mental health carve-out.
  • A for-profit mental health company with a carve-out contract.
  • The study also looked at fee-for-service care.

Levels of service reviewed included:

  • Inpatient.
  • Outpatient.
  • Residential treatment.

In analysis of the results, the "two-part model" described by Dr. Forrest was used to analyze first the probability of use and also the amount of use by users.

The results of the analysis showed that:

  • Children and youth involved in the juvenile justice and child welfare systems were more strongly affected by managed care.
  • Children and youth involved in the juvenile justice and child welfare systems experienced a greater decline in inpatient days and outpatient care.
  • The juvenile justice population experienced greater increases in the use of residential treatment.
  • Changes tended to be greater within the non-profit system as compared to the for-profit system.

Return to Contents

Patterns of Care, Expenses and Hospital and Emergency Room Use by Children with Chronic Conditions

James Perrin described a study that looked at the relationship of different "patterns of care" to service use and cost for children with chronic conditions, by type of condition or disease.

Patterns of care were defined as:

  • Primarily generalist care (no subspecialty visit in the prior year for the chronic condition).
  • Primarily specialist care (50 percent or more chronic condition visits were to subspecialists).
  • Co-managed (fewer than 50 percent of visits to subspecialists for the chronic condition).

The study subjects included all children receiving Medicaid SSI and a random sample of all other Medicaid recipients. Claims for the years 1989-1992 for four States (CA, GA, MN, TN) were analyzed.

The predominant form of care for all conditions except for heart disease was generalist care. The predominant form of care for patients with heart disease was co-managed care. There were substantial variations in rates of mean expenditures by condition.

A key methodological issue that the project addressed was determining physician specialty. Provider self-identification information was merged with an AMA Masterfile that included board status. The Masterfile was able to provide physician specialty for 65 percent of claims. For other claims, either the physician's self-identification was used, or procedure codes were used to determine specialty.

Another methodological issue was the ability to control for severity of cases. This ability is limited by the use of the ICD9 coding system, where a single diagnostic code may encompass multiple severity levels. Dr. Perrin discussed risk adjustment approaches, such as the Adjusted Clinical Group system, using administrative data.

Return to Contents

Current as of June 2000

Internet Citation:

Advanced Models and Methods, Part I. Role of Partnerships: Second Annual Meeting of Child Health Services Researchers. June 27, 2000. Agency for Healthcare Research and Quality, Rockville, MD.

Return to Overview
Proceed to Next Section


Page last reviewed June 2000
Internet Citation: Role of Partnerships: Second Annual Meeting of Child Health Services R: Advanced Models and Methods, Part I. June 2000. Agency for Healthcare Research and Quality, Rockville, MD.


The information on this page is archived and provided for reference purposes only.


AHRQ Advancing Excellence in Health Care