Skip Navigation U.S. Department of Health and Human Services www.hhs.gov
Agency for Healthcare Research Quality www.ahrq.gov
Archive print banner
Hospital Preparedness Exercises Guidebook

Public Health Emergency Preparedness

This resource was part of AHRQ's Public Health Emergency Preparedness program, which was discontinued on June 30, 2011, in a realignment of Federal efforts.

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: https://info.ahrq.gov. Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to www.ahrq.gov for current information.

Chapter 8. Evaluation

This section provides an overview of evaluating a hospital preparedness exercise in the following sections:

  • Overview: Evaluation of a Hospital Preparedness Exercise
  • Step 1: Planning and Organizing the Evaluation
  • Step 2: Observing the Exercise and Collecting Information
  • Step 3: Analyzing Information
  • Step 4: Developing the Draft of an After Action Report/Improvement Plan
  • Checklist: Evaluation
  • Useful Resources and Tools

Overview: Evaluation of a Hospital Preparedness Exercise

The purpose of an evaluation of an exercise is to assess the performance of participants and the emergency operations plan in order to determine the proficiency with which the hospital staff was able to carry out the tasks and demonstrate the desired capabilities and competencies, as well as the extent to which objectives were met. Evaluation of exercises is essential for identifying weaknesses and gaps, which is critical in improving and strengthening a health care organization's emergency management program.

Step 1: Planning and Organizing the Evaluation

Planning and organizing the evaluation of a hospital preparedness exercise normally takes place during the Design and Development stages. Important tasks that need to be completed in this step include:

  1. Assigning a Lead Evaluator

    This person is usually identified early in exercise planning and oversees the team of evaluators throughout the evaluation process. He/she helps to integrate evaluation planning into the exercise design process.

  2. Identifying the Evaluation Requirements

    These requirements are developed using the exercise scope and objectives. It may be useful to look at accreditation or Federal requirements as well. Also consider the tools, documentation, and personnel needed to carry out the evaluation process.

  3. Recruiting and Training Evaluators

    First, the requirements for evaluators should be identified (e.g., the number of evaluators, the type of expertise, and the time commitment needed). Then evaluators must be recruited, assigned, and trained to provide useful and constructive criticism and observations that are related to the exercise. Training may involve familiarizing the evaluators with the evaluation forms being used as well as the hospital's general operations. It may be helpful to do an exchange with another hospital or public health institution to obtain unbiased evaluators. Other potential evaluators include State or local emergency personnel not taking part in the exercise.

  4. Developing an Evaluation Plan

    The evaluation plan should include:

    • Exercise-specific information, such as the scenario and schedule.
    • Evaluator team information about organization, assignments, and location.
    • Evaluator instructions on what to do before and during the exercise.
    • Evaluation tools such as an exercise evaluation guide, the MSEL, and blank paper.


  5. Holding a Controller and Evaluator Briefing

    This should take place just before the exercise to provide updates and an opportunity to ask questions. In operations-based exercises, this may include a tour of the exercise area so that controllers and evaluators are familiar with where they are stationed to observe exercise play.

Step 2: Observing the Exercise and Collecting Information

Observing the exercise and collecting information take place during the Conduct phase of the exercise, and differ for discussion-based exercises and operations-based exercises.

Discussion-Based Exercises

For discussion-based exercises, there are often breakout sessions and small group discussions. Evaluators and data collectors need to be present at each of these small groups to get a full understanding of the thinking and decision-making during the exercise conduct.

Operations-Based Exercises

For operations-based exercises, evaluators and data collectors tend to focus on player actions and decisions. Normally, evaluators are situated in pre-determined locations so they can observe exercise play without creating a physical obstruction. Actions and decisions recorded for operations-based exercises include:

  • Who completed the action or task.
  • What the task or action was.
  • Where the task or action occurred.
  • When the task or action occurred (time or point in the schedule).
  • Why (the trigger).
  • How (the process of doing the action or task) things occurred.

During the exercise play, the evaluator also should be familiar with injects, messages, discussions, decisions, directives, and movement. This information may be presented in the briefing or contained within the Master Scenario Events List (MSEL). Having knowledge of the hospital's normal protocols and procedures is also useful, although specific medical knowledge is not necessary.

Evaluators may be provided with forms that allow them to rate performance on a numerical scale. They may also receive lists of objectives that should be completed during the exercise along with the tasks associated each objective. The evaluators would record whether the task was completed (fully, partially, or not at all) and record the time and other notes about the task. Evaluators should also record any problems or issues that arose during the exercise.

Hot Wash/Debriefing

Additional information may come from a hot wash (an open-forum discussion with exercise participants immediately after the exercise), participant feedback forms, and/or other debriefings with exercise participants. This may provide useful information that evaluators may have missed or misinterpreted and provide insight on how to improve the exercise or the emergency operations plan.

Tip: An actor debriefing may provide useful information about patient care and treatment during an emergency situation.

Step 3: Analyzing Information

After the exercise is conducted, the evaluation information is analyzed. Preliminary analysis begins in the controller/evaluator debriefing session. The information to be analyzed is gathered from debriefing sessions, evaluators' forms, participants' feedback forms and hot wash minutes. For hospitals, data may also include hypothetical clinical care outcomes for the "patients" of the exercise. At the post-exercise debriefing, controllers and evaluators have an opportunity to compare notes, begin to analyze findings including numerical measures such as performance ratings, and develop an account of the actions that took place during the exercise.

Data may be represented as narratives that show strengths and weaknesses during the exercise. HSEEP methodology suggests that, in the analysis, the evaluation team (e.g., the exercise planning team, exercise evaluators, and/or controllers) should identify the "root cause" for actions and tasks not completed in an exercise and develop recommendations based on this.

When conducting analysis of evaluation information, the evaluation team should consider:

  1. If objectives were met and to what extent.
  2. If tasks were successfully completed.
  3. What key decisions were made.
  4. Whether participants demonstrated they were adequately trained to perform the tasks and capabilities.
  5. If the actions of the players reflect the current plans, policies, and procedures.
  6. Recommendations for improvement.

Step 4: Developing the Draft of an After Action Report/Improvement Plan

After collection and analysis of information from the exercise, the exercise evaluation team should develop a draft of the After Action Report/Improvement Plan (AAR/IP). The AAR/IP includes a description of the exercise, strengths of exercise player response, areas of improvement that need to be addressed, and recommendations for implementing corrective actions.

According to HSEEP, an AAR/IP includes:

  1. Executive Summary—A brief overview that includes the scenario, the exercise objectives, what capabilities were tested, strengths, and improvement areas.
  2. Exercise Design Overview—This section includes the overall purpose of the exercise; the objectives, capabilities, and tasks tested; and a summary of the scenario and key events.
  3. Analysis of Objectives—This section examines the performance of the players on each capability/task, including strengths and areas of improvement.
  4. Appendix A: Improvement Plan Matrix—This contains an initial draft of the improvement plan.

Checklist: Evaluation

The following are some of the essential steps for the Evaluation phase:

Planning and Organizing the Evaluation

Before exercise conduct, the following tasks should be completed:

  • Assign a lead evaluator.
  • Identify evaluation requirements.
  • Hire and train evaluators.
  • Develop an evaluation plan.
  • Hold a controller and evaluator briefing (just prior to the exercise).

Observing the Exercise and Collecting Data

Data include:

  • Evaluators' observations from the exercise.
  • Exercise Evaluation Guides (EEGs).
  • Hot wash minutes.
  • Participant feedback forms.

Analyzing Evaluation Information

  • Condense information into narratives demonstrating exercise strengths and weaknesses.
  • Consider the extent to which objectives were met.
  • Determine root causes for actions and tasks not completed.
  • Develop recommendations for improvement.

Developing the Draft of the After Action Report/Improvement Plan

  • Write the draft of the AAR/IP.

Useful Resources and Tools

Below is a list of useful resources and tools related to evaluating an exercise.

Step 1: Planning and Organizing the Evaluation

  • Exercise Evaluation Guides

    Web site with a library of EEG templates developed by HSEEP that correspond to capabilities in the Target Capabilities List. They include guides for medical surge, mass prophylaxis, and medical supplies management and distribution, among others that are related to hospitals.
    https://hseep.dhs.gov/pages/1002_EEGLi.aspx (HSEEP)

  • Evaluation of Hospital Disaster Drills: A Module-Based Approach

    Provides valuable information and forms for evaluating hospital exercises, including scenarios related to decontamination, biological incidents, radiation incidents, and triage.
    http://www.ahrq.gov/research/hospdrills/ (AHRQ)

  • Evaluator Briefing

    Controller & Evaluator Briefing (HSEEP Vol. IV*, see Operations-Based Exercises > Conduct > Exercise Briefings)

Step 2: Observing the Exercise and Collecting Data

  • Emergency Operation Performance Evaluation Form Sample form that allows evaluators to rate performance of several tasks that need to be completed during an exercise.
    http://www.gnyha.org/374/File.aspx Exit Disclaimer [PDF Help] (North Shore University Hospital)

Step 3: Analyzing Data

Step 4: Developing the Draft of an After Action Report/Improvement Plan

  • After Action Report/Improvement Plan

    Templates of HSEEP-style After Action Reports with an Improvement Plan Matrix.
    Operations-Based Exercises: AAR (HSEEP Vol. IV, see under Evaluation > After Action Report)
    Discussion-Based Exercises: After Action Reports (HSEEP Vol. IV, see under Evaluation > After Action Report)

Note: These can be found as part of the HSEEP Volume IV Library: Sample Exercise Materials under Exercise Planning. https://hseep.dhs.gov/hseep_vols/default1.aspx?url=home.aspx

Return to Contents
Proceed to Next Section

 

The information on this page is archived and provided for reference purposes only.

 

AHRQ Advancing Excellence in Health Care