Skip Navigation Archive: U.S. Department of Health and Human Services U.S. Department of Health and Human Services
Archive: Agency for Healthcare Research Quality
Archival print banner

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to for current information.

Executive Summary

Expanding Research and Evaluation Designs to Improve the Science Base for Health Care and Public Health Quality Improvement Symposium

On September 13-15, 2005, AHRQ convened a meeting to examine public health quality improvement interventions.

Numerous gaps in health care and public health quality recently have been recognized, and studies on the effectiveness of strategies labeled health care and public health quality improvement interventions are a relatively recent development.1-7 Quality improvement (QI) in health care has recently been defined as "... identifying those activities that increase the rate with which practices known to be effective are applied to patient care."8 For this symposium, we defined QI interventions even more broadly, so as to include a range of strategies implemented in "real-world" settings for the purpose of expanding the delivery, reach, and impact of population-level, evidence-based health care and related public health interventions.

The goals for the symposium were:

  • To review a range of quality improvement interventions (QIIs) and the critical questions that arise in evaluation of these interventions, including basic questions of internal (does it work in the setting and for the condition in which it was tested) and external (will it work in other settings and conditions) validity.
  • To identify the strengths, weaknesses, and tradeoffs of alternative designs and methods for evaluating QIIs.
  • To achieve a working consensus about the range of traditional and innovative designs and methods that can be used to answer key QII questions.
  • To identify and suggest strategies to facilitate possible changes in funding mechanisms, review processes, research and publication standards, and research training that could help accelerate the development and spread of reliable QII research methods.

The format of the meeting was: 1) an opening night dinner with keynote remarks designed to frame the meeting; 2) presentations on quality improvement projects at a variety of levels of the health care and public health systems followed by commentaries and discussion periods; 3) breakout sessions in which all attendees were encouraged to make recommendations to improve quality improvement intervention research and evaluation, and 4) reports of the recommendations from the facilitators of the breakout sessions. This symposium was designed to be interactive and to elicit the input of the approximately 120 attendees, whose backgrounds spanned quality improvement, health care, public health, research, training, and patient advocacy.

Return to Contents


Introductory remarks focused on the need for quality improvement in health care, desirable characteristics of information from quality improvement studies, and that funding and performing randomized trials will not provide all the answers for QI because of the characteristics of quality improvement interventions:

  • The targets of QI interventions (QIIs) are not individual patients.
  • QIIs are complex and sometimes change over time.
  • The setting is an essential component of the question and QII.

A broad range of evaluation designs were described as potentially applicable to QIIs, including but not limited to, the randomized controlled trial, the group-randomized trial, the case-control study, the interrupted time series, and qualitative methods.

The quality movement, which is directed at the level of systems of care of communities and of the public's health, was characterized as the second phase of several recent important performance initiatives to reduce the "Knowledge-Performance Gap" in health care and public health. The first initiative was the practice of evidence-based medicine, which is largely directed at the level of care of individual patients. Drawing on elements of these initiatives, Batalden and Davidoff have developed an integrated model of improvement. The integrating concept that underlies this model is that improvement is fundamentally a learning process. The model is driven by three kinds of learning: a) scientific discovery, which is learning about "what is"; b) experiential discovery, which is "learning about learning" or, in effect, "learning about how learning works"; and c) experiential learning, which is learning "how to" do something. Experiential learning, to put it simply, is learning by doing. The importance of publishing was emphasized, and an announcement was made about forthcoming publication guidelines for QII studies.9

Return to Contents

Lessons Learned from Past QIIs

The symposium included a series of presentations on QIIs and lessons learned through them at four increasingly complex levels of the health care and public health systems. The first level examined was the clinical microsystem level. A clinical microsystem is defined as "a group of clinicians and staff working together with a shared clinical purpose to provide care for a population of patients."10 Hospitals, hospital units (e.g., intensive care units), primary care practices and other such entities can be considered clinical microsystems. The presentations examined QIIs for prevention in primary care settings and a QII for timely delivery of surfactant in neonatal intensive care units.

The second level examined was the health "plan" level. For this level, the presentation examined a series of QIIs performed within the Veterans Health Administration on collaborative care for depression.

The third example was a QII at multiple levels in the so-called "chain of effect" for systems change and quality improvement.11 For this level, the presentation reviewed lessons learned from the Improving Chronic Illness Care Evaluation (ICICE). The ICICE was an evaluation of the Institute for Healthcare Improvement's collaborative approach to quality improvement from organizational to clinical microsystem levels. In this intervention,"organization level" interventions included a requirement for the organizations to form interdisciplinary teams, and for senior leadership of the organizations to provide resources and a plan for disseminating the new Chronic Care Model-based care system. Changes actually implemented varied from organization to organization and, within organizations, at the clinic level, and, within clinics, at the individual patient-provider level.

The fourth level examined during the symposium was the state/regional level. For this level, the presentation examined the California state tobacco control and prevention effort begun in 1988.

Recommendations and observations addressed: 1) alternative research designs and methods for QII research 2) how to implement quality improvement (and implications for research), and 3) how to combine quality improvement with research. In addition, specific suggestions were directed toward funders of QI research.

The discussion on research design addressed the value and feasibility of randomized trials, as well as the types of data to be collected to understand more about the "whys" of program success or failure. The cluster randomized trial is a viable design which is applicable to studies of system-level change. However, some practices and health plans have been unwilling to participate in randomized trials. Studies within a single large health plan where the health plan response is one of the units of analysis cannot use randomized designs. In the absence of randomization, several aspects of design can help strengthen inferences about causality and generalizability, including: using an evaluation logic model; using a matched control group with a before and after design; collecting data from multiple sources; planning for and testing potential biases; and being sure that there is a solid evidence base for any clinical intervention as well as for clinical microsystem interventions.

In addition to measuring clinical outcomes, studies also need to measure behavioral and organizational changes that occur (intentionally and otherwise) as a result of QIIs. Currently, resource limitations can hamper the researchers' ability to collect data on variations in implementation across sites when sites are permitted to develop their own QII strategies. Knowing more about organizations will help the field understand how to tailor interventions to an organization's stage of readiness for change, and it will enable us to go beyond general statements, such as the importance of "leadership support and buy-in." New measurement tools may be needed.

Recommendations for combining QIIs with research focused on ways to increase both rigor and relevance: by working at multiple levels of a system, by developing participatory relationships that transcend single projects, by considering different ways of knowing (e.g., qualitative and quantitative data), by fostering shared learning among participants in research, and by pursuing development alongside research in QI. To facilitate more quality improvement engagement, more research is needed on how to entice practice leaders into QI studies. Funding agencies could help create research infrastructures in real-world settings and fund in shorter cycles to take advantage of natural experiments in QI.

The multi-level California tobacco control effort provided lessons for health care as health care QI is beginning to take place at levels beyond clinical microsystems and health plans. In the program, there were state-level policy changes (tobacco taxes) and a wide variety of grants for local programs. Thus, local adaptation was a built-in, essential feature of the program. Evaluations of program success depended on ongoing surveillance data supplemented by intervention-specific measurement and evaluation activities. For health care QIIs with these features, implementers and researchers should:

  • Maximize the design within the constraints.
  • Maximize learning from variation within the intervention.
  • Validate and then use existing surveillance measures.
  • Use multiple and differing measures of the critical phenomenon to increase one's confidence that there is a real treatment effect.

Return to Contents

Challenges to QI Research

Some of the challenges in QI research include the fact that people want to be able to isolate the effects of individual components within multifactorial interventions, but it is difficult for them to know where to invest resources to address this question. It also is necessary to strike a balance between structure and autonomy. The Institute for Healthcare Improvement's Breakthrough Series, for example, is a general QI process rather than a set of prescribed interventions. Centers want to have a menu of intervention choices that they can adapt to their needs.

Return to Contents

Overarching Recommendations

Numerous specific solutions for improving QI science were made during presentations and discussion of these studies or were generated by breakout groups tasked with addressing one of five areas: research designs, peer review and journal editing, health disparities, research training, and users of quality improvement research (stakeholders and evidence synthesizers). The solutions included:

  • Develop a taxonomy of QI for health care and public health.
  • Develop flexible intervention and evaluation toolkits that can be locally adapted.
  • Develop a transdisciplinary theory for health care and public health QI.
  • Address disparities in health in all health care and public health QI projects.
  • Build cumulative knowledge in QI.
  • Organize peer review panels to include transdisciplinary representation.
  • Develop funding mechanisms so that quick review and funding is available for the study of acute events and natural experiments in quality improvement and system change.
  • Fund a major dataset of QI research so that many types of questions can be asked.
  • Include partnership requirements in QII requests for applications (RFAs).
  • Work with the academic tenure and promotion system to develop viable career paths for QI researchers and implementers in academic institutions.

Return to Contents
Proceed to Next Section

Current as of March 2009
Internet Citation: Executive Summary: Expanding Research and Evaluation Designs to Improve the Science Base for Health Care and Public Health Quality Improvement Symposium. March 2009. Agency for Healthcare Research and Quality, Rockville, MD.


The information on this page is archived and provided for reference purposes only.


AHRQ Advancing Excellence in Health Care