Skip Navigation Archive: U.S. Department of Health and Human Services U.S. Department of Health and Human Services
Archive: Agency for Healthcare Research Quality
Archival print banner

This information is for reference purposes only. It was current when produced and may now be outdated. Archive material is no longer maintained, and some links may not work. Persons with disabilities having difficulty accessing this information should contact us at: Let us know the nature of the problem, the Web address of what you want, and your contact information.

Please go to for current information.

Section 3

An Agenda for Research in Ambulatory Patient Safety

Synthesis of a multidisciplinary conference to develop an agenda for research in ambulatory patient safety.

Experiences from Other Industries

Health care can learn from experience in other industries, particularly aviation: hypotheses, approaches, and some of the methods used to gain information, perform analyses, and implement measures to improve safety. Aviation is the example most often discussed, but there are others including nuclear power.

  • How one builds safer processes, including how to make best use of technology to improve safety, while taking advantage of knowledge of "human factors" that interact with processes and technologies to determine risk, and the value of teamwork in the cockpit, the intensive care unit, and the operating room (Reason 1990, 2001, Helmreich and Shaefer 1994, Welch 1997, 1998a, 1998b, Leape et al. 1999, Gaba 2000, Berwick 2001).
  • Methods of investigating and learning from accidents, incidents, errors, near misses (or near hits), how to work from an outcome back through the antecedent events and determine the human and systems factors that were involved. How to integrate with the analogous experience by physician liability insurance companies.
  • The experience in aviation with reporting of incidents and the use of a consolidated database of incidents for the industry as a whole.
  • How to deal with cultural barriers, though we are aware that there is much greater ability to effect change by control or force or regulation in aviation, for example, than in ambulatory health care.

The experience from other industries will need to be adapted to medicine to avoid resistance due to differences in the perspectives, assumptions, and culture between health care and these fields. Some of what works in other industries may not work in health care, even with adaptation.

Recommendation 3: Support research that builds on experiences from risk management activities of liability insurers, provider organizations, and integrated healthcare systems, and from other industries to:

  1. Understand risks and injuries in ambulatory safety.
  2. Where to focus efforts to further understand risks and to reduce them.

Return to Contents

Other Sources of Data on Ambulatory Safety

Providers, managed care organizations and pharmacy benefit management companies routinely collect large amounts of data. These data can be useful for identifying and quantifying the causes and effects of certain types of safety and quality problems. Such data may be useful for identifying prescriber adherence to best practices, patient adherence to prescribed therapy, and for connecting some processes and outcomes, especially when combined with medical event reporting systems specifically designed to address these problems.

The limitations of administrative systems, such as those designed for billing, quality assurance, and utilization management, are well documented. They may not reliably reflect some kinds of adverse events and errors, and the number and type of occurrences identified can vary according to source and method.

Billing systems are designed to capture activities and resources used to address specific services provided for specific conditions for individual patients. CPT and ICD-9 data can therefore help identify certain aspects of the process of care. Resources consumed can be quantified by using relative value units from the Medicare fee schedule (RVUs) as measures, and this approach can provide a means of comparing relative resource use across settings, specialties and individual providers. However, these data have limitations in detecting challenges to patient safety. We lack a generally accepted classification system or taxonomy for medical error. The ICD-9, DSM-IV, and CPT systems do not include a system for classification of error. Classification/taxonomy systems are being developed and tested at COPIC, several other malpractice liability companies and elsewhere. AHRQ will be sponsoring an IOM report that will focus on standardized vocabulary and coding for medical errors. It is a crucial step in attacking the problems surrounding patient safety, particularly in ambulatory settings.

Pharmacy benefits management companies (PBMs) companies process huge numbers of pharmacy claims and have built large databases useful for identifying problems in prescribing and use of medications. From these analyses, one can develop alerts for patients, providers, and pharmacies of potential or actual problems in medication prescribing and use.

The prevention potential of these systems is substantial if feedback mechanisms can be designed and implemented to directly inform providers of potential drug interactions, adverse reactions and identified risks of polypharmacy. Electronic medical records and electronic prescribing may facilitate better pharmacy information and feedback, especially in highly integrated delivery systems.

Administrative Data

Some databases of claims for payment can support population-based analyses that generate rates of incidents, injuries, and errors. For example, billing claims from a large self insured employer or a health plan enrolled population could support estimates of rates of selected kinds of incidents that can be shown to reliably generate claims (e.g., emergency care and hospitalizations for asthma, Vitamin K prescription to treat bleeding of patient on coumadin). Merging data from billing claims with additional information on medication use or laboratory results can provide a rich source of analyses. Databases can be generated to study a particular location (physician office), type of patient (post-myocardial infarction), service (obstetrics), or problem type (drug interactions), and to generate awareness and interest by clinicians.

Conference participants offered numerous examples of analyses of claims data and patient safety. Insurers are using claims data to identify examples of care that fail to follow recommended guidelines (e.g., failure to give beta blockers after acute myocardial infarction) with intent to intervene to improve practice, and several large insurers are combining their data to identify "over-prescribers" of antibiotics with plans to influence these physicians' antibiotic use (Kowalczyk 2001). Collaborative design and implementation of improvement efforts between insurers, providers, and provider organizations may help to improve safety.

Integrating insurance claims and pharmacy databases have yielded epidemiological information on aspects of the process of care that are known to be predictors of subpar outcomes. These data can be used to estimate rates of particular events that are known to increase risk of adverse events, such as asthmatics who don't get refills or are refilling too often (suggesting poor asthma control leading to overuse of the drug), patients on oral anticoagulants but not getting INRs (or claims for INRs), or post-MI patients who don't get beta blockers, patients receiving inappropriate combinations of drugs. Additional relationships can be examined if laboratory results can also be linked to claims and pharmacy data. This approach requires that one validate the relationship of risk factor to an adverse event.

However, the potential of administrative data will be limited because some kinds of adverse events and errors will not be reliably reflected in claims. According to Dr. David Bates, a conference participant, a much smaller fraction of adverse events and errors can be detected using claims data alone (using ICD-9 codes) than through chart review, patient interviews, or algorithms that draw on several sources (Karson and Bates 1999, Honigman et al. 2001a). The number and type of occurrences and incidents reported or identified can vary by source and method (Classen et al. 1991, 1992, O'Neill et al. 1993, Jha et al. 1998, Bates et al. 1998, Malpass et al. 1999, Thomas and Brennan 2000, Gandhi et al. 2000a, 2000c, Honigman et al. 2001a, 2001b). For example, one study of outpatient prescribing showed 18 percent of patients reported, when contacted by the study team, what they believed to be adverse effects to drugs while chart review showed just 3 percent (Gandhi et al. 2000a). Bates and colleagues have developed computerized detection systems for adverse drug events for both inpatient and outpatient care (Jha et al. 1998, Honigman et al. 2001a, 2001b, Jha et al. 2001).

Recommendation 4: Support pilot studies to determine the potential of administrative data, alone and in combination with other data sources, for research in ambulatory patient safety. Identify appropriate methods for using these data, and barriers to their use.

Though there are existing data that are available and suitable for further research, much of the needed data—particularly that to support analyses that are based on known populations—does not now exist, and will have to be generated through a variety of approaches. The conference participants identified several potential approaches.

Patient Surveys and Observing Care

Surveys of patients, physicians, nurses, and other staff can elicit information about actual and perceived risks, mistakes, injuries, and near misses. Surveys are useful to generate hypotheses for further investigation, and in principle, surveys can be used to generate estimates of population rates. Surveys must be carefully done to avoid low response rates and minimize bias, employing methods to obtain patient information that do not generate substantial concerns about liability and confidentiality, which are threats to unbiased responses.

One of the best uses of survey research techniques is for validation of research findings. There are consistent differences between errors reported in medical record reviews and those reported by patients. Analysis of these differences can give important insight into the magnitude problems of reporting error for patient safety in ambulatory settings.

There are the usual methodological questions about ability of persons to accurately identify and recall information like this, and additional considerations because of the sensitive nature of the information. For example, survey questions should be designed to neither alarm the patient nor de-emphasize risks, nor to induce suspicion of particular clinicians or organizations. Surveys requesting information from physicians about the kinds of adverse events and injuries they've seen happen to patients, both their personal mistakes and errors and those of others they've observed, will have to protect confidentiality and anonymity.

To complement methods focused on providers, tracking or following patients through care can help us understand how ambulatory care works (Stange et al. 1998, The DOPC Writing Group 2001). Structured interviews and surveys may also be effective methods to elicit this information. Such studies can highlight the potential for errors and adverse events associated with transitions and handoffs, while studies focused on one site of care may not. They also provide opportunities to understand how and where risks are generated by reliance on the patient to receive, understand, and act on medical information and recommendations. These studies would complement others that focus on particular sites (the physician's office) and particular processes (medication process, follow-up process), and case studies of occurrences and incidents that retrospectively reconstruct what happened to the patient and why.

Research Questions: What can be learned from patients' experiences in ambulatory care, and in transitions between ambulatory and inpatient care? What methods can produce reliable and valid information about handoffs and information transfer as well as capture patients' experiences as they view them (for example, are ethnographic methods useful)? For what kinds of data are patients—with and without training—reliable self-reporters, and for what data are trained observers required? Where can one use surveys, logs and diaries, or periodic interviews? How might one supplement patient reports with information from medical records and information from health care persons (and organizations) with whom the patient interacts? What patient groups would be most useful to study? Are there existing datasets generated for other purposes that could be used to identify occurrences and incidents?

Recommendation 5: Research should be conducted that assesses the perspectives of patients and families about ambulatory care, and characterizes the information they can provide about safety in ambulatory settings. Assess the reliability and validity of that information, identify methods that could ensure high rates of unbiased responses, and design potential studies that could yield population-based estimates of adverse events in ambulatory care and their causes.


Recommendation 6: Further research should be supported on the role of information technology to improve ambulatory patient safety, including computerized physician order entry and electronic medical records. These technologies should be evaluated within the larger contexts where they would be implemented and used:
  • The processes within which the technology will be embedded (e.g., the medication process).
  • The systems within which the technology will be used (e.g., the physician office practice).

Return to Contents

Reporting Systems: Learning about Safety and Improving Accountability

Before resources—attention, time, people, tools, money—can be most effectively brought to bear on system stabilization and improvement, problems must first be identified. Enough trusted (valid, reliable) data must be available to prioritize these opportunities. In the case of safety in health care, the "elephant in the room" for generations has been a neglected iceberg of ongoing preventable harms in health care delivery. Many barriers combined to create this situation. Some of these barriers are primarily internal to health professions and organizations (culture, training and teaching practices, lack of leadership for safety, scarce resources, meager clinical support infrastructure, poorly developed tools, etc). However, powerful social, political, and economic forces outside the health care system have had arguably more influence in helping to create and sustain the problem. Perverse financial incentives and a dysfunctional liability-compensation framework have supported a culture of silence and resistance in terms of dealing with safety failures that cuts across all layers in health care organizations.

Although a number of reporting systems have been operational for many years (e.g., National Nosocomial Infection Surveillance program, Institute for Safe Medication Practices, vaccination related events, the Adverse Event Reporting System (AERS) of the Federal Drug Administration, and several State programs), there are ways to improve upon them. Variations in nomenclature and in medical domains covered, lack of report detail, and growth of "electronic graveyards" are but a few of the issues. Mandatory systems have been honored in the breach, and have had unintended consequences as well (e.g., concerns of organizations reporting to the National Practitioner Databank have led to downcoding of reportable actions). Reports to some reporting programs, such as the JCAHO sentinel event databank are increasing (

The Institute of Medicine report (1999) recommended that each State should have a mandatory reporting system for medical errors. The National Academy for State Health Policy has completed a number of studies of State reporting systems (Rosenthal et al. 2000, 2001, Flowers and Riley 2001). They have found that 15 States have mandatory reporting systems for adverse events for acute care hospitals, but that they vary a great deal in design, definitions of reportable events, adequacy of funding, and whether they provide reports to the public or feedback reports to hospitals. They also vary in the legal and peer review protections they provide for reported data and for reporting entities. Of the 15 States that require hospitals to report, 13 also require reporting from free-standing ambulatory care facilities (Rosenthal et al. 2000).

New York State has a mandatory event reporting system called NYPORTS, established in 1985, in which all network hospitals within the State participate. Since 1998, all hospital data have been transmitted to the State through a Web-based data entry tool, and all sentinel event reports must include a JCAHO classification of underlying causes. In fiscal year 1999, the 286 participating hospitals reported 15,217 incidents, yielding an average of 625 incidents per 100, 000 discharges. These reports have been used to identify several areas of concern, including retained foreign bodies, the relation of diet to post-operative bleeding, and wrong-site surgery. Thus far, the system has been used principally for data entry, with limited capabilities for data analysis by individual hospitals.

A recent survey performed by the National Academy for State Health Policy revealed that only 15 States require mandatory reporting of adverse events from acute care and general hospitals, and they use a variety of definitions for reportable events (Rosenthal et al. 2000). The protection of confidentiality also varies from State to State. Significantly, three States did not offer protection from Freedom of Information Act requests and six were not sure. At this time, only eight States issue public reports. Incentives to report were limited: only three States offered feedback, and seven offered training in reporting. Under-reporting and inadequate resources were cited as the greatest areas of concern with reporting systems.

High profile government and foundation reports have begun to describe these issues in detail in the past several years (e.g., Institute of Medicine 1999, QuIC 2000). Information on reporting in health care and other risky, complex industries has become increasingly available both in the published literature and through many informal sources such as non peer reviewed publications, conferences, and policy forums. For the purposes of this conference synthesis, the major themes that organize the reporting dialogue are briefly presented in the context of improving ambulatory care patient safety.

Mandatory Versus Voluntary Reporting

There is a lively debate underway about whether reporting should be voluntary or mandatory (Institute of Medicine 1999, Barach and Small 2000, Cohen 2000, Runciman et al. 2001, Brennan 2001, Jencks 2001. As noted, the Institute of Medicine report (1999) included a recommendation that all States would have a mandatory reporting program, beginning with hospitals. A consensus was not apparent from discussions of these complex issues, but several interesting points were articulated. For example, if both mandatory and voluntary reporting are needed, they may be used for different purposes. And the architects of reporting systems must consider that, with few exceptions, accurate, in-depth reporting depends on voluntary behaviors on the part of reporters who cannot be effectively coerced. Coercion is also likely to induce resistance behaviors with unintended consequences such as downcoding of events, suppression of sensitive data, and proforma activities that consume resources but do not lead to improvements.

Voluntary reporting may work best for near misses and other threats to safety, for which the reporter's fear is small compared to adverse events with significant injuries, which are more likely to lead to lawsuits and claims. An often-overlooked benefit of voluntary systems is the possibility of systematically identifying examples of excellence, such as recoveries from events that could have had serious adverse outcomes, and disseminating those findings as case studies and as validated clusters that can lead to best practices.

It may be that an effective mandatory reporting system is necessary to meet regulatory requirements, to support deemed accreditation programs, or to build public confidence. One approach to mandatory reporting would identify a group of "never events" that all stakeholders can agree should never happen, and should be reported. Each must be well defined, and not difficult to identify and report, so that their incidence and prevalence can be easily measured and data analyzed and the results disseminated. The National Quality Forum (NQF) is moving to establish such a program (see "Serious Reportable Events in Patient Safety: A National Quality Forum Consensus Report" at Though the 27 events identified by the NQF have been defined for inpatient care, a number of them can relate indirectly to events in ambulatory care.

The relationship between reporting and liability will strongly influence the willingness of physicians and health care organizations to participate in a reporting system, whether voluntary or mandatory.

  • Studies by the National Academy for State Health Policy (Flowers and Riley 2001) show that the legal and peer review protections afforded by States to reporting entities and reported information vary in both form and extent. Policies for protection are less developed for ambulatory care than for hospitals. What are the peer review protections that need to be in place to support accurate, in-depth reporting, particularly for ambulatory care?
  • On a national level, could the scope of work of the Peer Review Organizations be extended to include the collection of near-miss data?
  • Legislation has been introduced in Congress to provide Peer Review Protection for errors reported to AHRQ; would this protection extend to entities participating in research and pilot projects linking reporting to actions supported by AHRQ?

The recently announced series of patient safety projects funded by AHRQ19 included an "Implementation Planning Study for the Integration of Medical Event Reporting Input and Data Structure for Reporting to AHRQ, the Centers for Disease Control (CDC), CMS, and the Federal Drug Administration (FDA)." The stated goals of this study are to:

  • Identify existing and developing data collection activities within the four participating HHS partners (AHRQ, CDC, CMS, and FDA), where they are located, how the information is stored (including specifications for data storage) and similarities and redundancies in the current data infrastructures.
  • Identify opportunities for future development of an integrated reporting system. Combined with efforts by the NQF and output from the safety grants coordinating center, both existing mandatory and voluntary reporting systems, and those under development, will likely converge on a number of standards.

The ideal reporting system may be said to have the following characteristics, which are a composite of recommendations from the NASA Aviation Safety Reporting System, the QuIC report, and other sources.

Characteristics of an Ideal Safety Reporting System

  • All stakeholders on board.
  • Intent and goals clear to all parties.
  • "Just" culture: reporters protected as much as possible from legal and other harms.
  • Accountability issues clear, focused, understandable—limited to reckless intent and violation.
  • Reporting includes both confidential and anonymous options.
  • Anyone can report.
  • Reporting made as easy as possible; multiple options.
  • Narrative as well as fixed field, choice selection data collection forms.
  • Field experts used for data analysis.
  • Feedback to all stakeholders, especially reporters, is rapid cycle and relevant.
  • Sustained leadership critical to protect core mission and values.

Effective, cooperative administration of the evaluation of existing reporting systems and the design of new systems will build trust and confidence. A new regulatory model should emerge, where regulators and others charged with strengthening accountability work collaboratively with the organizations that deliver care. Instead of a siege model in which embattled organizations spend scarce resources limiting damage and deciding what to report to whom in a patchwork, overlapping system, accountability can be strengthened by developing incentives and means for organizations to show how they are learning about improving safety, and with what results.

Sources of Data

Reporting systems usually focus on information from clinicians. Traditionally, incident reporting systems have depended primarily on reports submitted by nurses. Sparse research (e.g., JCAHO quality journal, Leape et al. 1995, Wakefield et al. 1996, 1999a, 1999b, 2000, perhaps, others) has documented the poor incidence capture rate of some of these systems. Isolated reports (Weingart; Petersen; Small) reinforce the notion that well designed pilot studies can capture detailed data from physicians about threats to safety and adverse events in a systematic manner.

In addition to relying on reporting by physicians, nurses, and other clinicians, information can also be gathered from patients, from clinical information systems, and both electronic and paper medical records. Jencks et al. (2000) describe an example of assembling information from billing data, medical records, and patient surveys for tracking selected aspects of quality of care for Medicare beneficiaries. A number of studies focused on inpatient and outpatient adverse events show that the occurrences and incidents reported or identified will vary by source, as noted above.20

Value of Emphasis on Near Misses

Collecting data on near misses offers less potential for litigation than does data on adverse events with injury.21 There is unique potential to capture maps of recoveries since injury was avoided (how was the event cascade recognized and controlled before injury occurred, or was injury avoided due to chance?) and use that information to systematically improve systems design, training, etc. The much larger number of near misses allows quantification and trending of types of events—a ratio of 1:300 for serious events to near misses (Heinrich 1941)—generating enough data to enable analysis of the power of interventions.22 The proactive approach, which attempts to eliminate threats before serious injuries happen, helps change culture in a positive direction towards learning. There may also be a reduction in hindsight bias, since it is easier to debrief and get more sensitive information about the causes of an event and to understand the interrelationships of potentially causative factors due to the lack of actual harm and consequently less guilt, fear, and shame.

Lessons Learned from Other Industries

There is much to be learned from experiences in other industries that can help design safety reporting systems and initiatives to improve safety for health care. However, there are differences between health care and these industries that will require thoughtful translation of lessons learned from them to health care.

Evolution of reporting systems. A number of complex, hazardous industries have experience with safety reporting systems over the past 30-40 years. Several trends have been observed. First, there has been a shift from anonymous to confidential sources of data, to increase validity and scope of information about events. This has been enabled by evolved legal protections, as well as through devices such as obtaining confidential data within a short window of opportunity after the event and then stripping the data of identifiers. Anonymous data is still valued, since the objective is to learn about threats to safety, and some events may be so sensitive, induce guilt to such a degree, or have such significant consequences that operators might not communicate the needed data without the protection of anonymity (Runciman et al. 2001). In some industries (e.g., commercial aviation), serious accidents may have become rare enough that near misses provide the best available data for learning and accountability. Over all, the emphasis has been on moving up the curve to increased detection and reporting of threats, anticipating decreased occurrence of harmful events as systems are made more reliable and safe.

Through many years of experience, the concept of "root cause" has evolved in these industries. The logic underlying root cause analysis is now complemented by a new model of disturbances in complex systems where many small, potentially inconsequential events may, if combined in the proper context, cause a disproportionately serious accident. While there are "upstream" factors leading to an incident (what Reason has labeled "latent factors"), the notion of root cause can create a false frame of reference that there are one or two ultimate, concrete causative agents or entities.23

Return to Contents
Proceed to Next Section

Page last reviewed December 2001
Internet Citation: Section 3: An Agenda for Research in Ambulatory Patient Safety. December 2001. Agency for Healthcare Research and Quality, Rockville, MD.


The information on this page is archived and provided for reference purposes only.


AHRQ Advancing Excellence in Health Care