Skip Navigation Archive: U.S. Department of Health and Human Services U.S. Department of Health and Human Services
Archive: Agency for Healthcare Research Quality
Archival print banner

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to for current information.

Overview of the AHRQ Project "Designing Consumer Reporting Systems for Patient Safety Events" (Text Version)

Slide presentation from the AHRQ 2009 conference.

On September 14, 2009, Michael T. Halpern made this presentation at the 2009 Annual Conference. Select to access the PowerPoint® presentation (400 KB).

Slide 1

Slide 1. Overview of the AHRQ Project "Designing Consumer Reporting Systems for Patient Safety Events"

Overview of the AHRQ Project "Designing Consumer Reporting Systems for Patient Safety Events"

Michael T. Halpern, M.D., Ph.D.
Senior Fellow
Division of Health Services and Social Policy Research.

Sept. 14, 2009
RTI International is a trade name of Research Triangle Institute.

Slide 2

Slide 2. Background


  • Since 1999, 25+ states have passed legislation or created regulations on hospital reporting of adverse events.
  • Many current event reporting systems are aimed at health care providers - don't facilitate input from patients and their families on experiences with care.
  • Such input is needed - may facilitate detection of patterns of patient safety events, highlight system flaws, and reveal information not necessarily observed or reported by providers.

Slide 3

Slide 3. Consumer Reporting Systems for Patient Safety Events

Consumer Reporting Systems for Patient Safety Events

  • Patients and family members are in a unique position to view continuum of care and identify gaps that may contribute to adverse events.
  • Systems including patient reports will complement information from health care providers collected through other reporting mechanisms such as PSOs.
  • Consumer reporting systems may improve understanding of medical errors and enhance methods for improving quality and preventing patient harm.

Slide 4

Slide 4. AHRQ Project: Designing Consumer Reporting Systems for Patient Safety Events

AHRQ Project: Designing Consumer Reporting Systems for Patient Safety Events

  • To develop recommendations for consumer reporting systems, AHRQ awarded a 2-year contract in Sept. 2008 to RTI International and Consumers Advancing Patient Safety (CAPS).
  • The project involves an iterative consensus-building process, including:
    • A technical expert panel.
    • Consumer focus groups.
    • Interviews with key stakeholders; and
    • An environment scan and literature review.

Slide 5

Slide 5. Project Framework - IDEALS Concept

Project Framework - IDEALS Concept

  • Based on the "Ideal Design of Effective and Logical Systems" (IDEALS) design concept (Nadler, 1967).
  • In this concept, a recommended system evolves through three stages:
    1. A theoretical system capturing a vision of an ideal system.
    2. An ultimate ideal system built on the theoretical ideal system but contains achievable operational and practical goals; and
    3. A technologically workable ideal system (TWIS).

Slide 6

Slide 6. Technical Expert Panel (TEP)

Technical Expert Panel (TEP)

  • Purpose: to provide input on features of ideal systems, technically workable ideal systems, and recommended systems for consumer reporting systems for patient safety events.
  • Panel comprised of 18 individuals selected to include experts in patient safety, event reporting systems, health care delivery, quality improvement, patient-centered care, and patient advocacy.
  • Recommendations generated through Delphi questionnaires and TEP meetings.

Slide 7

Slide 7. IDEALS Concept

IDEALS Concept

Diagram of a pyramid and with the following listed from top to bottom:

  • Theoretical Ideal System - 3-Round Delphi & Focus Groups with Consumers.
  • Ultimate Ideal System - 3-Round Delphi & Focus Groups with Consumers.
  • Technologically Workable Ideal System - TEP 1:Nominal Group Technique Environmental Scan and Leterature Review.
  • Complete Specification of TWIS - TEP 2, Focus Groups and Key Stakeholder Interviews, TEP 3.
  • Recommended System - External Peer Review and TEP Briefing.

Slide 8

Slide 8. Delphi Questionnaires

Delphi Questionnaires

  • Three rounds of questionnaires, with anonymous responses:
    • 1st round focused on 5 open-ended questions collecting information on functions, features, and characteristics of consumer reporting systems from an ideal perspective.
    • Second and third round questionnaires asked participants to rate and rank suggestions provided from the round 1 questionnaire.
  • Final round results described the ideal attributes of a consumer reporting system for patient safety events and formed the basis for the first TEP meeting.

Slide 9

Slide 9. First TEP Meeting

First TEP Meeting

  • First meeting held at AHRQ headquarters June 24, to discuss questionnaire results, provide more detail on design features of ideal reporting systems, and form initial set of recommendations.
  • Meeting facilitated using Nominal Group Technique (NGT), which allows TEP members to generate items on priorities individually and then share and discuss these with other TEP members.

Slide 10

Slide 10. Preliminary Consensus Points from First TEP Meeting

Preliminary Consensus Points from First TEP Meeting

  • Purpose of consumer reporting systems: improvement of health care and development of safer care.
  • System ownership: independent, to avoid conflicts of interest.
  • System governance: strong representation by consumers, but open to all stakeholders, using a commission model.
  • System design: allowing submission of consumer reports by multiple modalities (e.g., phone, email, fax, etc.) and providing real-time (24/7) response capabilities.
  • Level of operations: at multiple levels, from local to national or international.

Slide 11

Slide 11. Second, Third, and Fourth TEP Meetings

Second, Third, & Fourth TEP Meetings

  • Second TEP meeting (Dec. 2 & 3) will focus on parameters and consensus recommendations for TWIS, including potentially feasible approaches and barriers.
  • Third TEP meeting (webinar) and will focus on completion of TWIS specifications, final list of recommendations.
    • Recommendations from third meeting and the draft final report will undergo external review by at least three patient safety experts; recommendations will also be reviewed by consumer and caregiver focus groups.
  • Fourth TEP meeting (webinar) - TEP members will review feedback received from the external review process and develop the final list of recommendations.

Slide 12

Slide 12. Environmental Scan and Literature Review

Environmental Scan and Literature Review

  • Aim: explore technical feasibility and design elements for consumer reporting systems identified from Delphi questionnaires, first TEP meeting, and initial consumer focus groups.
  • Will focus on technically possible options for consumer reporting systems
    • Supporting elements.
    • Strengths.
    • Weaknesses.
    • Potential challenges for implementation.
  • Includes interviews with 9 key informants.

Slide 13

Slide 13. Focus Groups with Consumers

Focus Groups with Consumers

  • Six first-round focus groups will be conducted with patients and/or caregivers who have experienced near miss or adverse events.
    • Two focus groups held in Denver in June to provide ideas and input for developing consumer reporting systems.
    • Two more held in early September in Houston to provide additional ideas and inputs. Two additional focus group meetings scheduled for Boston.

Slide 14

Slide 14. Second Round Focus Groups with Consumers

Second Round Focus Groups with Consumers

  • Second-round focus groups will be held after second TEP meeting, to gather additional feedback on system design issues and recommendations from TEP.
  • Findings from focus groups will be shared with the TEP for incorporation in final recommendations.

Slide 15

Slide 15. Interviews with Key Stakeholders

Interviews with Key Stakeholders

  • 25 key stakeholders identified to participate in telephone interviews
    • Diverse group of individuals who can provide information on the design of consumer reporting systems for patient safety events.
    • 11 interviews completed to date.
    • Findings from interviews will be shared with TEP members without identifying sources.

Slide 16

Slide 16. Overview of Process

Overview of Process

Flow chart with four boxes:

  • Theoretical Ideal and Ultimate Ideal Systems:
    1. Three-round Delphi Questionnaires.
  • Technologically Workable Ideal Systems (TWIS):
    1. 1st TEP Meeting (in-person).
    2. Focus Groups with Consumers.
    3. Environmental Scan and Literature Review.
  • Completion of Specifications of TWIS:
    1. Interviews with Stakeholders.
    2. 2nd TEP Meeting (in-person).
    3. 2nd-round Focus Groups with Consumers.
    4. 3rd TEP Meeting (webinar).
  • Recommended Systems:
    1. External Peer Review.
    2. 4th TEP Meeting (webinar).

Slide 17

Slide 17. Dissemination Plans

Dissemination Plans

  • An Information Dissemination Plan Report will be prepared that will present outreach strategies to disseminate the project's key findings.
  • Key findings will be disseminated in:
    • Peer-reviewed journal articles.
    • News briefs in newsletters.
    • On web sites of major health care delivery organizations, patient advocacy organizations.
    • In presentations at conferences.
    • In brief publications in trade journal; and
    • In other media outreach activities.

Slide 18

Slide 18. Project Personnel

Project Personnel

  • AHRQ: Linda Greenberg, James Battles.
  • RTI: Michael Halpern, Amy Roussel, Linda Lux, Patrick Nerz, Katherine Treiman, Asta Sorensen.
  • CAPS: Marty Hatlie, Sue Sheridan.

Slide 19

Slide 19. For More Information . . .

For More Information ..

  • Please contact:
  • Thank you very much!
Current as of December 2009
Internet Citation: Overview of the AHRQ Project "Designing Consumer Reporting Systems for Patient Safety Events" (Text Version). December 2009. Agency for Healthcare Research and Quality, Rockville, MD.


The information on this page is archived and provided for reference purposes only.


AHRQ Advancing Excellence in Health Care