Chapter 5. Key Evaluation Questions

Assessing the Evidence for Context-Sensitive Effectiveness and Safety

Our framework is based on three key types of questions that describe three different aspects of patient safety practice (PSP) evaluation (Figure 7). 

We hypothesized that fully addressing context-sensitive effectiveness and safety would require studies addressing all three types of PSP evaluation questions, and that each evaluation question type would imply different methodological approaches. While an important result of this project is a more expansive view of the characteristics of ideal studies of PSP implementations, effectiveness questions are often assessed using experimental or quasi-experimental designs; implementation experience questions frequently use a pre-post design; and adoption, spread, or sustainability questions would require, at a minimum, observational or descriptive designs. Questions on context sensitivity could be framed in terms of each type of evaluation. The information on effectiveness; implementation experience; and adoption, spread or sustainability in relationship to context would be part of the full picture of final judgments on context-sensitive effectiveness and safety.

The conceptual framework we defined is in line with the multi-modal approaches often used in the PSP field. Saying that an effectiveness study should have information on the process or completion of implementation relative to context, for example, will be meaningless if recognition is not also given to the multiple designs indicated by the different types of questions. In other words, we are not interested in someone's opinion of the implementation process or of how context affected it; we are interested in data. The randomized trial that reports an overall PSP effectiveness result without data on context often cannot be effectively applied. Similarly, knowing that small size practices did worse on PSP outcomes than larger ones (context) may be somewhat useful, but it is not likely to be as useful if data have not been collected to understand what went wrong in the smaller practices. For example, was the PSP fully implemented in these practices? Did small size practices that succeeded better in implementing the PSP show greater impact on expected outcomes? Thus, combined approaches are favored that simultaneously seek rigor in precision regarding effectiveness while also reporting key contextual features and, if possible, assessing contextual inferences on the outcomes of effectiveness, implementation experience, and adoption or spread.

Within the timeline for this project, we focused most of our efforts on the first evaluation question—effectiveness and how it is influenced by context—and did not at all address the evaluation questions about adoption and spread.

Page last reviewed December 2010
Internet Citation: Chapter 5. Key Evaluation Questions: Assessing the Evidence for Context-Sensitive Effectiveness and Safety . December 2010. Agency for Healthcare Research and Quality, Rockville, MD.