Skip Navigation Archive: U.S. Department of Health and Human Services U.S. Department of Health and Human Services
Archive: Agency for Healthcare Research Quality
Archival print banner

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to for current information.

Reports from the Heterogeneous Breakout Groups

Expanding Research and Evaluation Designs to Improve the Science Base for Health Care and Public Health Quality Improvement Symposium

On September 13-15, 2005, AHRQ convened a meeting to examine public health quality improvement interventions.

The core planning group met to consider the recommendations from the homogeneous breakout groups, and sorted them into these three topics:

  1. What can we do to get our act together as a field (such as creating toolkits, guidelines, methods)?
  2. Communicating with/engaging others who are not us.
  3. "Growing the tent"/Recruiting others outside our tent.

Two heterogeneous breakout groups were asked to address the 1st topic, one to address the 2nd topic, and two groups to address the 3rd topic.

Dr. Mercer reported back on topic 1: What can we do to get our act together as a field (such as creating toolkits, guidelines, methods)?

This group noted the importance of a taxonomy. It is important to define terms, including what quality improvement (QI) is and what it is not. However, we have struggled with the issue of definitions in other burgeoning fields and do not want to exclude anyone by not incorporating their discipline's perspective so there needs to be a lot of caution when proceeding on a taxonomy. We want to be careful about how we define and use terms and we need to know who needs to be at the table. When we are working with a group in a messy, real world situation, we need to define what we mean so we can come to the table with ideas, but we do not want to do it in a way that is exclusive.

We noticed that Deming, for example, has not yet been mentioned in the course of this meeting, and this reminds us that we need to have a sense of history and to keep track of advances in the field to date.

There is a certain messiness when you are doing a project in the real world. One needs to bring in all involved parties at the earliest stage possible, such as bringing in evaluators and users during project planning. We need to think about how the information is going to be used and think of users' needs at the start. We need to think about internal and external validity from the beginning of a project. We may not be able to address both from the beginning, but we do need to think about this context from the outset of a study.

Other areas we discussed included noting that trainees are important. We need to develop future researchers. Some might be from institutions with no background in QI and we should think about how they could use the symposium participants as mentors. We need to give them real world, messy experience in projects.

People need to be able to measure disparities in all their projects.

If projects are to be successful, one needs to consider demands, interests, and incentives of all people involved, including researchers and participants.

We kept coming back to the need to expand our tent to work with others. We need to think about how we fit with others. While we want to define what we are doing, we do not want to be exclusive and separate ourselves from health services research and other areas. We need to think about how we work with people in other fields. Funding agencies need to work together collaboratively and to participate in the process, but not drive the process. Maybe we could expand this to the Centers for Medicare and Medicaid Services and other organizations as well.

Dr. Fine reported back on topic 1: What can we do to get our act together as a field (such as creating toolkits, guidelines, methods)?

In our group, our theme was a focus on research, and one important point that was made was that the definition of health research across the Department of Health and Human Services (HHS) should include what we are talking about at this conference.

We discussed improving or adapting the peer review process for funders and funding mechanisms such that the process better serves QII and is more user-friendly. For example, peer review panels should include transdisciplinary representation We discussed that there is a need to find a timely way to provide funding for acute events to facilitate their study. We noted that the Department of Veterans Affairs (VA) has been wrestling with this with some successes.

When an agency wants to expand the type of research it covers, there should be special training to educate study section members in their new broadened role.

We need to continue to encourage cooperation across the HHS funding agencies. We noted that the VA is a model for QI and AHRQ understands the issues, but some in the group believed that this is less true at the National Institutes of Health (NIH).

We should make sure funding can go to anyone who is capable, not just academics. The disease-specific orientation of NIH institutes is a dilemma for this field which cuts across diseases.

We touched on theory—small, medium, and large—and suggested that it would be nice to have funding for theory development by itself, and not just as part of another project.

There were discussions about toolkits, methods, and Web sites to communicate better about methods and instruments that work well and as a resource for finding them. The VA has one (, but we need more. Electronic journals are another way to address that need, especially in that they can accept longer papers. One issue for guidelines for publications is to describe the details of an intervention and its context.

One negative point was raised: there were several votes for not having a registry of studies. Participants felt that at this point in the field, a registry would not be a useful thing.

Dr. Francis reported back on topic 2: Communicating with/engaging others who are not like us.

We are interested in engagement with end users, but we had few end users at the session and in the meeting, so we need to take this dialogue and this kind of conference to end users. Leif Solberg is working to organize just such a meeting.

A second goal that we could set for ourselves is to stop whining. We have been talking about QII from a scarcity paradigm as if we are not understood, there are not enough resources, and there are not enough people doing this kind of work. However, if you look at people who have been successful in doing QIIs, such as administrators and people in operations, they did not bemoan the vagaries of gransmanship or the lack of academic cachet, but instead sought out the people who had the evidence as well as those willing to support the work. We need to engage with health systems and community leaders and thereby access the people and resources available. Thus, we need to change from a scarcity metaphor to an abundance mentality.

The third issue we discussed was a short-term to mid-term goal. The increasing availability of electronic health information through electronic medical records and other data repositories raises concerns about subject protections and the need to work with institutional review boards (IRBs) to ensure accountability. In the past, when we considered something QI, as opposed to research, it was believed that IRB and Health Insurance Portability and Accountability Act (HIPAA) requirements did not pertain. Current IRB structures and processes are not well adapted to QII research, and there is considerable discussion on designing mechanisms more appropriate to such work. Ongoing work on this topic is being funded by the Hastings Center.

For mid- to long-term goals, we discussed that we need new multivalent models of research in which one is doing complex interventions and including end users in study design and even on study sections.

Dr. Green reported back on topic 3: "Growing the tent"/Recruiting others outside our tent.

As our broad goal, we looked to expanding the tent in two directions: to end users and to a wider range of disciplines and fields. We identified short-term objectives under each of those goals. Also, we have some intermediate objectives, such as incorporating more of the continuous quality improvement/short loop feedback perspective in schools of public health and other training programs, and more emphasis on process or mediator variables in courses on evaluation, as well as better use of natural experiments. We identified a second intermediate objective: the need to embed QI into health care and public health. Strengthening translational research may be a means to that end.

We identified two major long-term goals: 1) expanding the tent to a wider range of, and more deeply involved, end users, and 2) expanding the tent to more disciplines to incorporate their models, theories, methods, and experiences into the tent. Regarding expanding the tent to include end users, this has to happen early (rather than waiting until the tent is in final order) and we should include non-governmental organizations (NGOs) as partners in developing the tent. We identified 12 specific short-term needs regarding working with NGOs:

  1. The need for engagement of users in developing a taxonomy of questions and prioritizing those questions.
  2. The need to develop meaningful collaborations.
  3. The need to embed users in all phases of the research process.
  4. The need for support to develop collaborations.
  5. The need for partnership requirements to be built into requests for applications (RFAs).
  6. The need to develop standards, guidelines, and criteria for what constitutes a genuine partnership.
  7. The need for follow-up and maintenance with partnerships after research.
  8. The need to involve end-users in the peer review process.
  9. The need to develop training about partnerships for partners.
  10. The need to understand how to get along with academics.
  11. The need to address how this field will work with the academic promotion and tenure system.
  12. The need to assess "what is in it for me?" for each partner.

Regarding the need to incorporate a wider range of disciplines, this group had three examples of short-term objectives:

  1. We want to involve evaluators whose work has been in other sectors.
  2. We want to include more on QI in HSR training programs and in health policy and health management programs.
  3. We had an inconclusive discussion about including health insurers in partnerships. Maybe there is a conflict of interest there.

Dr. Dougherty reported back on topic 3: "Growing the tent"/Recruiting others outside our tent.
We chose part of the "expanding the tent" charge to focus on how to develop transdisciplinary theory for QI (but included other topics in our discussion). Everyone agreed that we do need a transdisciplinary theory and practice for this. But the question was, "how do we develop that?"

  1. The group recommended having conferences with a focus on users' needs (such as what does an MD need to know from an anthropologist, a psychologist, or an organizational specialist, and how can we bring their disciplinary languages together?).
  2. We examined who should be in the tent. We started making a list of individuals, which seemed overwhelming, so we turned to listing major organizations that should be included. People listed social science lobbying groups and the NIH's Office of Behavioral and Social Sciences Research and suggested looking to these groups to identify specific people. We need to provide some incentives for getting people to participate, and we need to look at how research teams are structured to make sure that all participants are recognized as peers.
  3. Form follows funding so it is up to funders to require transdisciplinary teams. However, funders are influenced by others so we need to reach out to others who can be influential. We may need to make some sort of campaign to make QI important to people, following the model of the IHI 100,000 lives campaign. We might want to bring in AcademyHealth. The tent should not just be focused on the researchers and the evaluators.
  4. We need to engage users to bring in a multidisciplinary perspective and we need to be able to use natural experiments.
  5. Agencies can be a role model for transdisciplinary work. The Robert Wood Johnson Foundation (RWJF) is a role model for including a lot of different disciplines who each have something to add.

Plenary Session and Discussion

Thomas Chapel, M.A., M.B.A. (Facilitator)
Senior Health Scientist, Office of the Director, Office of Strategy and Innovation, Centers for Disease Control and Prevention

Mr. Chapel asked a few people in the planning committee to comment on the reports back session regarding the big issues about which we would like to hear cross-talk or underlying or fundamental issues that we have not gotten to yet. He asked them to mention if there are any big underlying questions that have not been addressed in the conference so far.

Dr. Orleans emphasized that one of the most important cross-cutting issues the field faces is disparities, and this should be addressed in every one of the initiatives with which we move forward. Another issue that we should continue to be addressed is consumer involvement or patient involvement. Also, Dr. Francis's point about linking to operations experts is an important one.

Dr. Wagner summarized that the recurring theme of the conference is that there seems to be an emerging field but there does not seem to be an infrastructure in the funding community and in the academic community to support this emerging field.

Dr. Dougherty hoped to discuss how do we organizationally build a field, move this field, and have a systems perspective. It is clear that health care QI has been slow to adopt a systems perspective, and that has been reflected at this conference. Each person may have thought about what they might do to build the field but not necessarily how they might work with others to build the field. She asked for feedback from the user's perspective. The VA is a model, perhaps because within the VA one is aware of the patients and caretakers every day in that they are part of the VA system. AHRQ is different because it is a research agency. The AHRQ mission is to improve health care for all Americans, and AHRQ needs to work with the people who are doing that work through health plans. Dr. Dougherty would like to hear from people in that field to find out their needs.

Dr. Green noted that the business model for continuous quality improvement continues to be used in the business world, but other than continuing discussion of short-loop feedback cycles, the push within the field of QI for respectability and the pull from outlets for publication result in a tendency for the RCT to emerge as the driving methodology in this area, and that emphasis pushes against the short-loop feedback cycle as the area of emphasis in this field. Dr. Green appealed for ways to keep that on the agenda in the field. Dr. Green was happy to hear that Dr. Brian Mittman and Dr. Martin Eccles are going to include information on process measures, intermediate variables, and moderator variables in articles in Implementation Science.

Dr. Francis indicated that while we have been talking about developing a new field, it is imperative not to overlook the importance of generalism. Creating a separate specialty and fragmentation is part of the academic paradigm, but it is one of the barriers to doing QIIs well in health systems and public health venues. Overspecialization may make a person less capable of having an impact where practice occurs. We do not want to become so overspecialized and jargon-ridden that we lose contact with others.

Mr. Chapel opened the floor to comments from all participants.

One participant noted that in regard to engaging the user, it is important to be observant and participatory, and to allocate energy to listening. It is also important to be aware what operations look like and to spend time in the clinical microsystem and the operations context. This participant saw this meeting as the first part of a conversation.

One participant wanted to discuss whether NIH views translational research in the same way as other entities do. The old concept of translational research as "bench to bedside" and getting drugs to market is very different from what has been discussed at this meeting. Is there a need to better interface with that effort in order to accomplish more?

A participant commented that the notion of a field coming together is very important but very tricky. There is no clear definition of what the field really is. The definition does not depend on the methods used, because they are drawn from other fields. The field could be defined by its goal of improving health care, and this suggests a marriage of patient safety and quality improvement worlds. We should think through the semantics of whether this is quality improvement research, research to improve quality, and whether there are implications of the semantics. Regarding whether there are any voices we have not heard from today, he also noted the absence of manufacturing, meaning those who use quality improvement in a variety of areas and have introduced it to their fields, from this meeting. We should try to learn from those outside the health care field.

A participant noted a paper by Thomas Gilmore of the University of Pennsylvania on field building in philanthropy. Dr. Gilmore uses the example of groups paying more attention to end of life care and makes the point that how the moral imperative is presented is an important part of energizing a field. We should think about what is the moral imperative that has been expressed throughout this meeting. One example is winning the tobacco wars. We are selling ourselves short if we do not make the case for the importance of health care QI in terms of the moral imperative. Analogies to business are useful, but may be overdrawn, particularly for state departments of health or nonprofit organizations where the constraints are different from those in businesses. Sometimes analogies can harm an effort because, for example, one does not run a grass roots effort the way one runs General Motors. We should think about pattern matching in order to determine where such analogies are helpful and where they are not helpful.

A participant remarked that, as a consumer advocate, her goal was to improve health outcomes for everyone. She cautioned against putting too much emphasis on trying to develop this area as an emerging field. She felt that that will take care of itself.

A participant thanked the planners for including consumer organizations and asked participants to seek more partnerships with consumer organizations in designing plans and programs because groups such as hers are advocates. As such, they can advocate for what makes sense to them in having quality health care. They can be champions for this effort if they are included as partners.

A participant advocated including operations people in these efforts. He described an example from his health plan of how partnerships can work. At HealthPartners, they have a new internal grant program with approximately $250,000 to spend each year on projects requiring partnership between a researcher and an operations person. They have a fast-track review process in which the applications are reviewed and funded within a month of submission.

A participant noted that this effort may not require a new field. What we are talking about is trying to produce predictable change in human behavior in a variety of settings and under a variety of constraints. If we conceptualize the field as the study of human behavior, we have something that already is a scientific field.

Return to Contents
Proceed to Next Section

Current as of March 2009
Internet Citation: Reports from the Heterogeneous Breakout Groups: Expanding Research and Evaluation Designs to Improve the Science Base for Health Care and Public Health Quality Improvement Symposium. March 2009. Agency for Healthcare Research and Quality, Rockville, MD.


The information on this page is archived and provided for reference purposes only.


AHRQ Advancing Excellence in Health Care