Chapter 14. Results: Needs for Future Development

Assessing the Evidence for Context-Sensitive Effectiveness and Safety

As described in our methods, we developed a list of 13 domains for future research. We then surveyed our technical expert panel (TEP) and queried which of these domains could be considered high priorities for future research. All 22 TEP members responded. Six domains received majority support as high priorities for future research (as bolded in Table 8).

We now expand on these survey results with a summary of the TEP discussion of these topics.

1. Developing and validating measures of patient safety culture. Discussion at the panel meetings indicated that several technical experts considered patient safety culture to be the overarching important construct. This view may explain why patient safety culture received majority support as a high priority for future research, whereas research on leadership and teamwork measures did not. Specific suggestions for future research included:

  1. Developing validated measures of cultural adaptability to change.
  2. Assessing the potential distinction between a culture of safety, a culture of excellence, and organizational culture.
  3. Establishing connections between aspects of patient safety culture and patient outcomes or processes of care.
  4. Assessing correlations between measures.

Additional comments that we received can be summarized as "we think teamwork and leadership are important," "several measures are currently available," and "the most important thing at this point is for people to use them so we can start building some evidence about this construct."

2. Developing criteria and recommendations for what constitutes "reporting the intervention in sufficient detail that it can be replicated." More precise criteria for how PSP interventions should be described warrant additional research. In particular, the guidance described here, along with that provided by Standards for Quality Improvement Reporting Excellence (SQUIRE) and the National Quality Forum (NQF), needs to be evaluated. Doing so will help determine which PSP elements are necessary to describe in order to evaluate whether the PSP is truly effective. This also will help maximize the possibility of successful PSP replication with similar outcomes. Further research could also evaluate the effect of applying these draft criteria regarding PSP descriptions on the quality of PSP projects and published articles. Clearly, thoroughly describing PSPs also can help readers determine the relevance of an evaluation study to other PSPs or other contexts. For example, if a PSP requires an individual behavior change such as hand-washing, then knowing in detail what the intervention is may help readers of the study assess whether the given results are relevant only to hand-washing interventions or if they could be applied to other types of PSPs requiring individual behavior change. Knowing the details of the intervention also could help readers of the study determine how much the success of the PSP implementation depended on contextual issues (e.g., organization or teamwork). 

3. Understanding the important items to measure and report on for implementation. Experts consider having comprehensive information about implementation key to being able to replicate a PSP. However, little empirical evidence exists about what makes a description of the PSP adequate for reporting. It is critical to assess what implementers need to know if they are to be able to implement or adapt an intervention in their own settings. Most experts considered "understanding the important items to measure and report on for implementation" to be related to or even the same as "reporting the intervention in sufficient detail that it can be replicated." This view suggests that the distinction between "the intervention" and "the implementation" may be an arbitrary line, and that ideal evaluations of PSP interventions need to consider the implementation as part of this intervention.

4. Developing a theory-based taxonomy or framework with which to describe and evaluate key elements of interventions, contexts, and targeted behaviors. Although the current project made a promising start on meeting this need, progress in this area will require additional development to produce a taxonomy that is both sufficiently broad based and flexible enough to be widely useful. Issues to be considered include whether a taxonomy is the preferable way to proceed, or whether a more useful strategy might be to create an explicit methodology that researchers could apply to specific problems and contexts. Yet another approach might be to devise an "assessment framework." Some experts sounded cautionary notes on this topic. They reported that outpatient PSP research may be too new to apply a taxonomy at this stage. They also reported that a single "unified" taxonomy may not be sufficiently flexible for diverse PSPs, and multiple taxonomies may be needed in any case. The countervailing view to these cautionary notes was that the field would not be well-served by having a proliferation of taxonomies. Instead, they reported, what is needed is a coherent, sufficiently comprehensive taxonomy that can accommodate the challenges of the subject.

5. Refining a framework for assessing the strength of a body of evidence. We did developmental work on an adaptation of the GRADE and EPC systems for assessing the strength of evidence across studies of a PSP. This work warrants further development.

6. Generating empirical evidence that the contextual factors identified in this project influence the success of the PSP. We acknowledge that most of the recommendations in the report have a thin empirical evidence base, which simply reflects the relatively immature state of research in this still relatively young field. Building a stronger evidence base will help future efforts at refining the recommendations presented here.

Additionally, the TEP acknowledged unintended consequences and rare events are important but not a high priority at this time. The issue of how to assess PSPs for context-sensitive beneficial outcomes in situations where the outcome is not rare was considered to be the more important priority.

Continuing the TEP's Efforts

We additionally queried the panelists about whether they would be willing to continue participating should AHRQ decide to continue this kind of methodological development work. All 22 panelists responded affirmatively, and many volunteered enthusiastic and laudatory comments about the process.

Of the list of "high priority" items, those we judge most likely to be fruitful for additional work using this TEP are:

  • Developing a theory-based taxonomy with which to describe and evaluate key elements of interventions, contexts, and targeted behaviors.
  • Developing criteria/recommendations for what constitutes " reporting the intervention in sufficient detail that it can be replicated" and the related topic of understanding the important items to measure and report in implementation.
  • Refining a framework for assessing the body of evidence about a PSP.

All three of these topics were discussed as part of this project, and this report contains the results of those discussions. But each of these, alone, could have been the focus of an entire 2-day TEP meeting, rather than the 1 or 2 hour allotment they received. Often we found we had to "move on" from a rich and insightful discussion in order to try and discuss all of the topics on this project's agenda. Additional development of the preliminary ideas is needed, and this TEP is both an appropriate group to pursue this and already engaged.

Page last reviewed December 2010
Internet Citation: Chapter 14. Results: Needs for Future Development: Assessing the Evidence for Context-Sensitive Effectiveness and Safety . December 2010. Agency for Healthcare Research and Quality, Rockville, MD.