Chapter 11. Description of Ideal Evaluation Methods: Assessing for Possible Harms

Assessing the Evidence for Context-Sensitive Effectiveness and Safety

All interventions can cause harms. Often these harms are unexpected adverse events. The methods of detecting unexpected adverse events are much less well developed than are the methods for detecting benefits, even for traditional assessments of efficacy or effectiveness of pharmaceuticals or devices. Long after having been studied for efficacy and receiving approval for use, unexpected associations have been discovered between COX-2 inhibitors and myocardial infarction, atypical antipsychotics and death in elderly patients, certain bisphosphonates and jaw osteonecrosis, and drug-eluting coronary stents and late restenosis. Therefore, our TEP judged an assessment of possible unintended adverse events to be an important criterion for an ideal evaluation of a PSP. They offered the following suggestions for how this might be accomplished:

  1. Before PSP implementation, spending designated time with the developers, the implementers, and organizational staff to brainstorm about what could go wrong is key (this exercise is sometimes known as a "pre-mortem" discussion). This can include a careful "walk through" of the logic model, assessing where problems may arise. More formal methods for prospective risk assessment could be used. For instance, failure modes and effects analysis has been used to anticipate changes (including what could go wrong) related to technology implementation.1-2  For potential adverse events foreseen by this process, evaluators can incorporate into the evaluation measurements of these potential harmful processes and outcomes and then include such information in their quantitative analysis.
  2. For adverse events that are truly unexpected, meaning that no amount of pre-implementation planning could account for them, direct interviews with organizational staff responsible for implementing the PSP would be one way to assess for their occurrence. Educators can then retrospectively ask questions such as, "How did this really work?" or "Did anything go wrong?" and then follow up appropriately. Another method is for the implementers to keep a log or diary of issues related to implementation and use of the PSP. The list of issues can then be analyzed to identify those issues that were anticipated and those that were not.

References for Chapter 11

  1. Wetterneck TB, Skibinski KA, Roberts TL, et al. Using failure modes and effects analysis to plan implementation of smart intravenous pump technology. Am J Health Syst Pharm 2006; 63:1528-38.
  2. Bonnabry P, Despont-Gros C, Grauser D, et al. A risk analysis method to evaluate the impact of a computerized provider order entry system on patient safety. JAMIA 2008; 15(4):453-60.
Page last reviewed December 2010
Internet Citation: Chapter 11. Description of Ideal Evaluation Methods: Assessing for Possible Harms: Assessing the Evidence for Context-Sensitive Effectiveness and Safety . December 2010. Agency for Healthcare Research and Quality, Rockville, MD.