Panel 2: Broad-based Systems Approaches
Testimony of Dale Bratzler, D.O., Chair, Medical Affairs Section, The American Health Quality Association
The first National Summit on Medical Errors and Patient Safety Research was held on September 11, 2000, in Washington, DC. Sponsored by the Quality Interagency Coordination Task Force (QuIC), the Summit’s goal was to review the information needs of individuals involved in reducing medical errors and improving patient safety. More importantly, the summit set a coordinated and usable research agenda for the future to answer these identified needs.
Individuals were selected by the Agency for Healthcare Research and Quality (AHRQ) to testify at the summit as members of the witness panels. Each submitted written statements for the record before the event, documenting key issues that they confront with regard to patient safety as well as questions to be researched. Other applicants were invited to submit written statements.
Disclaimer and Copyright Statements
The American Health Quality Association (AHQA) represents a national network of 38 health care Quality Improvement Organizations (QIOs), 37 of which serve as Medicare peer review organizations. Most QIOs also work for Medicaid and other public and private purchasers. QIOs work continuously with providers and practitioners across the nation to improve health quality.
QIOs are often both the users and the implementers of research findings. For example, QIOs have relied on AHCPR clinical practice guidelines or evidence-based practice reports to construct clinical pathways, and then worked with providers and practitioners to help implement them in practice.
Most debate over the IOM report on medical errors has focused on reporting of adverse events and how to hold providers accountable. The QIO community has been more interested in the portion of the report which focused on the causes of errors and ways to reduce and prevent them. While more information would be useful, it is our contention that it is not necessary to wait for error reporting systems to be established and refined before making substantial progress toward patient safety.
AHQA believes it is critical that AHRQ research agenda on patient safety include studies that identify the most effective methods for facilitating changes in caregiving processes. Our research questions fall into two broad categories:
- What are the best methods to identify opportunities for clinical quality improvement?
- What are the best methods to facilitate action to improve quality by practitioners and providers?
Identification of Opportunities for Improvement.
Providers and Practitioners are the Key to Identifying Opportunities for Improvement.
While consumers do often identify opportunities for clinical quality improvement, such opportunities are generally identified either through the activities of health care providers and practitioners, or by entities and individuals external to the caregivers. As an example of the latter, the Health Care Financing Administration (HCFA) has identified six clinical topic areas (breast cancer, diabetes, heart attack, heart failure, pneumonia, stroke) which are targeted by a coordinated national quality improvement program involving the Medicare Peer Review Organizations (PROs). As an organization decidedly outside of the caregiving process, HCFA was nonetheless able to identify specific, measurable opportunities for improvement, such as the frequency of administration of aspirin and beta blockers for appropriate heart attack patients, because of the large amount of evidence that has been amassed by caregivers and researchers on the need for and underutilization of these therapies. In the future, AHRQ or the National Quality Forum might conclude after similar research that certain other clinical areas (e.g., patient falls, nosocomial infections) deserve focused attention in a national patient safety initiative.
Although a substantial body of literature may establish the need for action to improve care in a clinical area, the work of identifying the root cause of a failure to engage in optimal processes remains the unique responsibility of practitioners and providers. In addition, for many types of errors or near misses there is no substantial literature base which identifies the opportunity for improvement. These must be found by providers and practitioners in the caregiving setting, through review of information maintained for routine purposes (e.g., the medical record) or databases created specifically for the purpose of identifying errors and near misses (e.g., incident reports, or a patient fall registry). These methods are all complemented by the knowledge gained by the day-to-day interactions between caregivers and patients.
Question: How much data is necessary to identify meaningful opportunities for improvement?
One research topic deserving of AHRQ’s attention is gauging the return associated with increasing levels of investment in gathering data and information on preventable adverse events or near misses. This is an important resource question for providers, practitioners, and government agencies. It is also significant because delays in achieving clinical quality improvement may occur if time and personnel are diverted to gathering more data than is needed to begin improvement work. How much evidence is enough?
The answer to this question is likely to be specific to clinical processes. Many error prone clinical processes (e.g., preventable adverse drug events) have been identified and are well-studied, with numerous articles appearing in the clinical literature. In addition, private organizations such as Medicare Peer Review Organizations (PROs) and the Institute for Safe Medication Practices (ISMP) become familiar with problems and solutions in scores or even hundreds of hospitals, and share lessons learned in these institutions with others. By relying on these reports and expertise, physician and provider quality improvement efforts can largely skip the analysis of incident reports, medical records or other data, and focus directly on whether critical procedural safeguards are in place.
Even in the comparatively well-researched area of adverse drug events (ADEs), there is ongoing debate as to when enough is known about clinical processes to forgo original investigations at each caregiving site. Some ADE researchers suggest a focus directly on the presence or absence of procedures and safeguards in place in health care organizations (e.g., standardized labeling for tubing for instances in which multiple simultaneous infusions must be administered), rather than invest time and money in looking for instances of errors, because there are no valid predictors of preventable medication errors (Bates et al, Arch Intern Med 1999; 159:2553-2560). Others have identified patient-specific factors (e.g., declines in kidney or liver function) associated with medication errors which should be used to identify opportunities for system improvements (Lesar et al, JAMA 1997;277:312-317). Choosing between these approaches is probably unnecessary, and hospitals could employ either approach.
For clinical processes and care settings where the important work of identifying universal error-prone processes and generally useful solutions has yet to be done, providers and practitioners face the question of how much digging is needed to arrive at an adequate understanding of the problem. A single "sentinel event" may be very informative, yet it can be risky to rely on a single isolated event for clues about system failureseven one involving a bad patient outcomebecause that event may have arisen for atypical reasons that say little about the caregiving systems in place at the institution or clinic. Unavoidably, and possibly often, it will be necessary to distinguish a systemic problem from an idiosyncratic or special cause by undertaking a review of multiple incidents or near misses. So the question remains: how much effort should be invested in identifying instances of preventable adverse events and "near misses"? One way to begin to answer this question is to examine the utility of various sources of information on adverse events and "near misses."
Incident Reports. An obvious place to look for adverse events is incident reports, but the overwhelming majority of errors are not documented in incident reports. Research conducted at LDS Hospital in Salt Lake City found 731 ADEs occurred during an 18 month study period, during which time 9 ADEs were documented in incident reports at the institution, about 1.2 percent of the actual number (Classen et al, JAMA 1991;266:2847-2851). Cullen, Bates, and others found 54 adverse drug events generated only three incident reports in five patient care units at a large northeastern teaching hospital. Twenty-six of these ADEs were judged serious or life-threatening, but only 2 of these cases resulted in an incident report. These AHCPR-supported investigators found other similar reports in the literature. (Cullen et al, Joint Commission Journal on Quality Improvement 1995;21:541-548, 1995). AHRQ might usefully fund investigations to determine whether meaningful quality improvement can proceed from the narrow factual base of incident reports, particularly since so much political capital is being invested in compelling hospitals to transmit these reports to external organizations for analysis.
Medical Record Review. Medical record review may produce significantly more evidence of preventable adverse events than incident reports. Indeed, most of the literature on adverse events is based on what can be determined from a careful review of the medical record, including both of the major studies cited by the National Academy of Sciences, Institute of Medicine (IOM) report published in November 1999. An example of an operational use of this data source is the Medicare Peer Review Organization (PRO) program, which will utilize a systematic review and abstraction of approximately 1.5 million medical records over the period 2000-2002. Abstracted data from these recordsgenerally not provider specifichave been used very effectively to persuade caregivers of the existence of opportunities for improvement, as well as to measure improvement that occurs.
In addition, medical record review may be a less expensive alternative for smaller and rural hospitals that cannot afford more comprehensive operational system reforms, such as installing an integrated automated medication ordering system. The Leapfrog Initiative, launched by some of America’s largest corporate purchasers of health care, recognizes the budgetary limitations of rural institutions in this regard by exempting them from their call for automated medication ordering systems. In lieu of sophisticated systems present in health care organizations such as Intermountain Health Care (IHC) and the Regenstrief Clinic, it may be feasible to manually search medical records for the presence of the same indicators of ADEs that were used by the computer system at IHC’s LDS Hospital (e.g., use of diphenhydramine, naloxone, or ordering of a digoxin level) to identify an error that may have occurred.
Medical records as a source of data on preventable adverse events and near misses suffer from some important limitations, however. First, while patient charts are very helpful in documenting "errors of omission" such as beta blockers not prescribed for ideal patients, they are not as useful in identifying "errors of commission," where sensitivities to liability may discourage documentation of the care provided to patients who suffer the consequences of adverse events. In addition, "near misses" may be difficult to discern from medical records. O’Neil and her colleagues noted that even documented adverse events were missed by the trained record reviewers in their study; "near misses" would presumably be even more difficult to identify (O’Neil et al, Ann Intern Med 1993;119:370-376). Finally, even if medical records can be searched manually for "errors of commission," employing indicators of preventable adverse events such as those developed at LDS Hospital for use in querying their automated medication order system, these record reviews are expensive (approximately $60/record for the high-volume abstraction being performed for the Medicare PRO program). Still, it must be noted that some Medicare PROs find hospitals willing to abstract medical record data for Medicare patients (and for all patients in some states) as part of quality improvement projects with the PRO, despite the high cost of reviews.
Physician Reporting. Direct requests to physicians, soliciting reports of instances of adverse events, were successfully employed at one large urban teaching hospital. Physicians (house staff) identified approximately the same number of adverse events as was found through medical record review, but the cases found by these two methods were often not the same. Physicians reported 89 adverse events, while 85 were found in medical record review, but only 41 events were common to both methods (kappa = 0.52). Physician reports sometimes provided more relevant context for adverse events than medical records. Another benefit of this approach is that physicians tended to identify adverse events judged preventable about twice as often as record review. In addition, physician reporting, which was stimulated mostly by email in the O’Neil study, was a good deal less expensive than chart review: $15,000 compared with $54,000. The authors of this research noted two subjective factors that influenced the outcome of their project. Physicians felt that they were being asked to exercise their judgment and contribute to quality improvement, which helped spur more reporting. Also, the researchers’ status as trusted outsiders apart from the hospital organization, who assured confidentiality for reports, appears to have been an important factor in securing so many useful reports from physicians. Medicare PROs enjoy this same status with most hospital providers, and may be well-positioned to receive such data for analysis and feedback to providers and practitioners. The utility of this approach in institutions lacking a teaching affiliation remains an unresolved question. (O’Neil et al, Ann Intern Med 1993;119:370-376)
Special Databases. Development of specialized or topic-specific databases is an effective strategy for solving the problem of underreporting of Incident Reports. The Oklahoma Foundation for Medical Quality, a quality improvement organization holding the Medicare PRO contract for Oklahoma, created a special database to receive reports of patient falls from hospitals. Over 1100 reports were received from over 40 hospitals and some long term care facilities. The PRO analyzed these reports and conducted educational sessions with provider representatives. This approach was well-received by the provider community. Incident reporting to an independent database of this kind has all the benefits of Incident Reports, including details of the adverse event not usually found in the medical record, and makes it possible to identify several opportunities for improvement in the caregiving process. This approach may be most suitable for use in a small number of focused clinical areas, given that it adds some reporting burden to health care workers.
The most comprehensive approach to identifying errors and "near misses" would be to ask providers and practitioners to augment their incident reporting systems with medical record reviews, solicited physician reports, and special focused reporting systems, since all of these methods produce some benefit and there is some evidence that each may identify additional cases. Unfortunately, the cost of doing so would likely be prohibitive. Providers, practitioners, and all who depend on them to improve health quality (consumers, health plans, employers and other purchasers) would benefit from AHRQ-funded research to define the circumstances (including type of facility, location, nature of the clinical process being examined) under which each method of identifying opportunities for improvement could be optimized.
Question: What is the effect of mandatory reporting on successful identification of improvement opportunities?
The goal of mandatory reporting of "medical errors," as expressed in the IOM report, is to increase the volume and quality of information available to the public regarding errors. Voluntary reporting is presented in the report as the model to be employed by providers and practitioners seeking evidence of preventable adverse events to reduce the incidence of these in the future. We could find no studies of the impact of mandatory reporting on voluntary reporting. Two possibilities seem likely. Mandatory reporting may cause hospitals and others to (1) create new and sophisticated systems for monitoring performance and responding to preventable adverse events, or (2) become fearful that these data will be used in a way detrimental to the provider’s interest, resulting in curtailing of even voluntary efforts that might inadvertently identify incidents that must be reported to external authorities. Because errors and "near misses" are difficult for anyone outside of the caregiving setting to observe objectively, and a large proportion of these instances and system problems are difficult even for health care workers to identify in the course of day to day labors, the very notion of "mandatory" reporting must be questioned. People are unlikely to work hard to find evidence of problems if this evidence will be used against them; yet, active "mining" for this information in medical records, incident reports, physician reports, and special databases is precisely what is needed to identify opportunities for improvement.
The experience of the leaders in the Medicare PRO program is that confidentiality must be assured or adverse event data and information will not be forthcoming. The PRO experience further suggests it is more effective to emphasize persuasion and provide a safe learning environment in which providers and practitioners are willing to engage in the difficult and time-consuming work of error identification, analysis of root causes, and changes in work processes. To more fully examine this question, AHRQ should sponsor research to (1) ascertain whether there is any evidence that mandatory reporting improves the effectiveness of quality improvement activities, and if so, under what circumstances; (2) compare the measurable improvement associated with mandatory reporting systems, compared with voluntary learning-based systems, and (3) determine whether mandatory reporting has any discernable influence on the quality or effectiveness of voluntary reporting systems where they coexist.
Facilitating Action to Improve Quality by Practitioners and Providers
Question: What is the most effective way to disseminate information about error-prone processes and related improvements to providers/practitioners?
The preponderance of published articles suggests that physicians respond best to clinical suggestions when presented by other health professionals in face to face encounters, such as "academic detailing" (Avorn and Soumerai, NEJM 1983. 308:1457-63; Schaffner et al, JAMA 1983. 250:1728-1732; Ray et al, JAMA 1986. 256:2536-2539) or use of physician "opinion leaders" and "educational influentials" (Stross and Bole, Arthritis Rhem 1980. 23:846-849; Stross et al, Am Rev Respiratory Disease 1983. 127:739-746), or a combination of the two methods (Lomas, et al, JAMA 1991. 265:2202-2207.) The studies cited here were all conducted using a control group. The experience of the Peer Review Organizations (PROs), which do not conduct experimental research but seek to improve clinical performance as a part of the day to day operations of the Medicare program, both confirm the findings of the literature and suggest other options that have been successful.
Both onsite and remote outreach to physicians and hospital personnel have been successful in PRO quality improvement projects. AHRQ should evaluate the circumstances under which success has been achieved in projects of both types. Advantages of onsite contact which are worthy of further examination include the ability for PRO physicians to speak directly to attending physicians, as well as the opportunity to customize suggested clinical pathways to the local circumstances of a rural or small hospital institution. Advantages of remote contact include the ability to provide summary scientific and clinical data to every hospital and physician in a State, and frequent, inexpensive contact with hospital staff assigned to quality improvement, who can follow up with physicians and other caregivers onsite. Other factors are likely to be cited by providers and practitioners if they were queried as to why they accepted and adopted PRO recommendations for changes in clinical processes.
The Cooperative Cardiovascular Project (CCP), which began as a four State pilot project and then, based on its great success, became a national project involving every PRO in every State. Under the CCP, PROs gathered data about performance of health care organizations in treatment and secondary prevention of heart attack, and fed back this data in a non-punitive, confidential environment to hospital personnel and physicians. A CCP Best Practices Working Group was formed from the senior staff of several PROs. This Group then identified the most effective hospital projects under the CCP, in 191 hospitals in 33 States, and followed up with a series of interviews to determine which barriers and methods for overcoming them were found most effective by the hospital personnel involved in these exemplary providers. The most frequent and successful interventions undertaken by PROs and their hospital partners were identified as follows:
- Hospital staff education.
- Development or revision of clinical pathways and standing orders.
- Ongoing data collection throughout the project.
- Development of interdisciplinary teams.
- Identification of a strong physician champion for the quality improvement project.
- Translation of clinical and scientific data into reliable and valid measures applicable at the local level.
These methods, which were employed to improve patient safety by increasing the use of evidence-based therapies in the care of heart attack patients, are readily extended to other clinical quality improvement topics and programs. (The Cooperative Cardiovascular Project Best Practices Working Group. Joint Commission Jrnl on Quality Improvement 1998. 24:480-490.)
Question: Does the effectiveness of error reduction efforts vary by size, location, or affiliation with a larger organization (e.g. integrated health system, hospital chain, physician group practice, or physician management group)?
Larger health care organizations or organized systems of providers and practitioners can be expected to have better information systems, and to have personnel specifically tasked with error identification and reduction. These attributes almost certainly facilitate quality improvement. Yet because fewer people and system hand-offs are required it is easier to achieve change in the systems of a small institution, such as a small rural hospital. It may be more feasible to uncover preventable adverse events in larger health care organizations, but this advantage may be offset by the significantly greater barriers to changing the processes that have been developed within a large and complex organization. Ambulatory physician practices may suffer from difficulty in accessing medical records or diverting office staff to support record review or any of the other means of identifying opportunities for improvement.
Strategies employed by AHRQ or HCFA to promote quality improvement may need to be adjusted for these different care settings, owing to their very different infrastructure. For example, many PROs have found that providing carefully constructed electronic medical record abstraction tools is sufficient to secure the cooperation of hospitals in record abstraction and analysis. These PROs consistently report that the act of abstracting one’s own records and developing one’s own sense of the opportunities for improvement at an institution is among the most powerful of tools motivating quality improvement. Yet, where small hospitals and physician practices are concerned, it is possible that external entities such as PROs must directly perform some of the tasks normally undertaken by the staff of larger hospitals. AHRQ should probe further into these potential tradeoffs and seek examples of strategies to address the barriers to quality improvement which may characterize different settings of care.
Question: Is a New National Patient Safety Infrastructure Needed?
Congress is presently considering a wide variety of legislation which would address patient safety. Most bills would create a reporting system, under which hospitals and other providers would be obliged to transmit incident report information to an external organization for evaluation and oversight purposes. These bills are drafted with two major purposes in mind: ensuring accountability of providers, and creating publicly available data on the nature and extent of errors (though provider and practitioner and patient names are not to be made public). As we have discussed, the paucity of data in incident reports limits the effectiveness of these reporting schemes. However, acknowledging those constraints, aggregated data from thousands of hospitals may well identify broad patterns and error-prone clinical processes that are present in many if not most provider settings. In addition, some of these bills also include provisions that aim to accomplish the three major patient safety tasks of identifying opportunities for improvement, creating practical solutions, and sharing these with providers and practitioners nationwide.
The question arises, however, as to whether new organizations should be created to gather, analyze, cull personal identifiers, and report to Federal authorities the knowledge gleaned from the combined reports. More importantly, is there a need for a new set of organizations to serve as catalysts for quality improvement with health care providers and practitioners?
AHQA respectfully submits that the answer to this question is "No." Congress created a national network of Peer Review Organizations (PROs) specifically to monitor and improve the quality of health care received by older and disabled Americans, simultaneously providing extensive confidentiality protections against disclosure of personal or institutional identifiers. Since the program was created in 1984, Federal investments in training and infrastructure have easily exceeded one billion dollars. This national infrastructure of PROs is capable of performing these tasks ably, and without jeopardizing the trust of providers, practitioners, or consumers. We recommend that both Congress and the AHRQ look to this existing capacity and reject calls to reinvent this particular wheel.
On behalf of the American Health Quality Association, thank you for organizing this important and timely summit. Our members look forward to working with you to design a research agenda that puts more and better tools into the hands of providers and practitionersthose with the most to offer in bringing about safer patient care in America.
Current as of September 2000
Return to Testimony Directory
National Summit on Medical Errors and Patient Safety
QuIC Home Page
Department of Health and Human Services