Making the Health Care System Safer
Second Annual Patient Safety Research Conference (continued)
Research-to-Practice Success Stories
The second panel focused on the lessons that can be learned from cases where research findings have been successfully incorporated into the everyday practice of medicine.
Using Data Warehousing Technologies to Understand Medication Error Processes
Dr. C. Andrew Brown reviewed his organization's efforts to use data warehousing technologies to understand and improve the reporting of medication errors.
Several years ago the University of Mississippi Medical Center (UMMC) faced a number of problems related to the reporting of medication errors. The data collected via the Application Service Provider was flawed, difficult to work with, and did not meet the needs of the facility or researchers. UMMC had little control over what was collected, the information could not be transferred or sustained, and it did not meet HIPAA requirements. Most important, perhaps, key data points were missing, including information on the age of patients (present in only 45.5 percent of reports), gender (48.4 percent), and reason for the error (49 percent).
To address these problems, UMMC added a module to the institution's existing system that provided anonymous reporting, risk analysis, clean data, data validation via needed workflow, and the ability to export data to a data warehouse that could support the study of medical errors at the institution. The warehouse linked to other data on economics, demographics, and clinical information, including ICD-9 and CPT codes. To facilitate the reporting of errors as they occur, UMMC also developed a Web-based form page with a direct link to this add-on module. The system could be accessed via the desktops of any of the more than 30,000 personal computers on the UMMC system, with users being given unique identifiers to log on as well as universal codes that allowed authorized individuals to log on without identification.
The new UMMC system allowed for a variety of data inputs to the warehouse (e.g., an IV hotline, occurrence reporting, patient interviews, medical records, clinical data, economic data) as well as routine reporting of errors for research and performance feedback and improvement. It has dramatically improved reporting, workflow, and data quality:
- The number of medication errors reported has increased by 85 to 90 percent, even with limited marketing of the system.
- The average time from error reporting to sending the error to risk management has fallen from three to four weeks to six hours.
- Data quality has dramatically improved; virtually all demographic data are now reported, compared to 50 percent or less before.
- Reporting of—and the reactions of caregivers to—inpatient and outpatient medication errors have changed dramatically.
- Near-miss medication errors that cannot be linked to a patient are now routinely reported.
- Data are updated for both hospital and ambulatory care patients every 15 minutes (rather than hourly, as occurred before), ensuring that data are available when clinicians need it.
The system also allows UMMC to engage in a new type of analysis that identifies the root causes of medication errors and near misses. This analysis found that four processes predict approximately 90 percent of medication errors while five processes were responsible for roughly 85 percent of near misses. This same technique also allowed UMMC to understand the different roles that nurses, physicians, and pharmacists play in medication errors and near misses, as well as the reasons for these errors. For example, human factors (e.g., inadequate communication) seem to explain the bulk of pharmacist errors while inadequate drug knowledge is an important factor in many nurse errors.
The Endoscopic Sinus Surgery Simulator
Marvin P. Fried, M.D., F.A.C.S, professor and university chairman of the Department of Otolaryngology at the Albert Einstein College of Medicine of the Montefiore Medical Center, reviewed progress to date in using an endoscopic sinus surgery simulator (ES3) to reduce medical errors. One of three principal investigators of the study, Dr. Fried, highlighted the expert, national team involved in the effort, which involves the collaboration of a number of well-known medical centers throughout the country.
The ES3 was developed in 1997. Based on flight simulation technology, it offers a virtual reality simulator that can be used to train individuals with various levels of experience. The simulator attempts to provide an experience that is as similar as possible to real surgery. The simulator can be programmed for routine procedures and can also simulate hazards and complications.
Medical students, otolaryngology residents, and attending physicians use the simulator after completing a training curriculum. Dr. Fried described six sub-projects that are a part of the study. Each is designed to test various hypotheses about simulation training, as outlined below:
- Expert surgeons will perform better on the ES3 than will less experienced residents, who will perform better than will novice medical students. This hypothesis has been confirmed.
- A correlation exists between an individual's psychomotor ability, video-spatial and perceptual ability, and performance on the ES3.
- Strong performance on the simulator leads to strong performance in the real world. This critical hypothesis (testing the "predictive validity" of ES3) is being examined through a study that will compare the performance of residents trained on the simulator to that of a control group receiving no training.
- Optimal performance on the simulator will occur with proximal feedback.
- Performance on the ES3 will correlate with performance on cadaver dissections.
- Initial training on the MIST/VR (another type of simulator) will optimize training on the ES3.
The ES3 project has enjoyed a number of successes thus far.
- A metrics conference identified the various core metrics and components that make up the surgical procedure. These metrics were recently published.
- A manuscript currently in progress calls for permanency of ES3 training for medical students.
- A novel Web-based database has been created.
- As noted, "concurrent" validity has been established—that is, individuals with more experience tend to do better on the ES3 than do novices.
- Preliminary data supports ES3 as an effective teaching tool. Once an individual has trained on the simulator, they retain their level of performance.
In addition to these successes, collaborations have been formed with the US Naval Medical Center, New York University Medical Center, and the New York Eye and Ear Infirmary to serve as additional ES3 data collection sites. A collaboration has been formed with another AHRQ grant recipient to use the simulator to assess the effects of sleep deprivation on surgical resident's skills. In addition, the project team expects to collaborate with two companies (Lockheed Martin and Mentice) to bring the next generation of simulators on line, with a goal of making these simulators a commercial product. Finally, the ES3 has received favorable attention from the media. Looking ahead, there are plans to enhance the virtual reality features of ES3 and to increase its technological sophistication, including the development of applications to allow the inputting of patient-specific data, which would allow surgeons to conduct realistic "practice" or "rehearsal" surgeries before the actual procedure is performed.
Video Record as a Tool for Patient Safety
Colin F. Mackenzie, M.D., F.C.C.M., serves as director of the National Study Center for Trauma and Emergency Services and as professor and vice chairman of the Department of Anesthesiology at the University of Maryland, Baltimore, School of Medicine (UMBSM). He discussed the value of creating and reviewing a video record of procedures as a tool for patient safety, particularly with respect to addressing human factors that contribute to medical errors. The value of video records lies in the ability of providers to review their own care, and to have subject matter experts review care as well.
Dr. Mackenzie shared results from video reviews of esophageal intubation procedures at UMBSM. Patients undergoing such procedures often become the victims of medical errors due to incorrect tube placement, which can frequently lead to brain damage or death. Errors can occur when physicians forget to complete one or more of the 12 preparatory tasks for intubation. Review of video records revealed three common problems:
- Carbon dioxide (CO2) analysis is delayed after intubation during emergencies.
- Clinical exams are delegated to someone other than the intubator.
- Communication failures occur related to the clinical findings from the chest examination after intubation.
Based on the video review, the medical center developed a task algorithm that is designed to ensure that critical tasks are not omitted. In addition, circuitry (that costs only $0.25) has been added to allow immediate CO2 analysis. These changes, made in 1993, have resulted in tremendous improvements in safety. During 10 years of follow-up there have been no cases of undetected esophageal intubation among the more than 14,000 intubations performed. (This figure compares to a 2.5-percent rate among patients who come to the hospital after being intubated in the field.) In 11 years of recording, there have been no legal issues, no employment-related issues, and no cases where clinician confidentiality has been infringed.
Effects of Extended Work Hours on ICU Patient Safety
Charles A. Czeisler, Ph.D., M.D., professor of medicine at the Harvard Medical School and chief of the Division of Sleep Medicine in the Department of Medicine of Brigham and Women's Hospital, shared the results of a study designed to determine the effects of extended work hours of residents on patient safety in the intensive care unit (ICU). He began by reviewing the physiologic factors that determine alertness and performance in humans:
- Biological time of day (circadian phase): The body's internal clock determines when certain hormones are released as well as body temperature, which in turn govern the ability of the brain to perform.
- Number of hours awake: Not surprisingly, individuals who have been awake for extended periods of time (e.g., 24 hours) feel more fatigued.
- Nightly sleep duration: Individuals who consistently do not get enough sleep over a period of several weeks will experience a decline in performance.
- Sleep inertia: Most humans do not reach peak performance for 5 to 15 minutes after being awakened. Performance can be seriously degraded during this "waking-up" period.
Unfortunately, residents often find that these physiologic factors work against their performance. Their circadian phase is often misaligned; they are frequently asked to stay awake for long periods of time; they often do not get enough sleep in a given night; and they are frequently expected to perform critical tasks within minutes of waking up.
The impact of sleep deprivation on patient safety has been known for 30 years. Yet the demands being placed on physicians and surgeons in training continue to result in sleep deprivation. To address this issue, AHRQ and the National Institute of Occupational Safety and Health (NIOSH) each launched an initiative in 2001 to study the impact of work hours on safety. AHRQ sponsored two studies—one examining the impact of working conditions on quality of care and the other evaluating the effects of extended work hours on ICU patient safety. NIOSH funded studies to evaluate the impact of work schedules and extended work hours on worker and intern safety. Dr. Czeisler described the ICU study in greater detail. Its goals were as follows:
- To comprehensively quantify medical errors in the ICU and coronary care unit (CCU).
- To assess the effects of the intervention for interns during their ICU/CCU on-call rotations on medical errors across the whole ICU/CCU.
- To assess the costs and benefits of the intervention.
The study will monitor 24 interns annually, observing them over their three-week ICU and CCU rotations. Performance will be evaluated using direct continuous observation of medical errors, voluntary and solicited reports, computerized adverse drug events detection monitors, chart reviews, and computerized order entry decision support tracking. Alertness will be monitored at least once per week through continuous ambulatory monitoring of EEG/EOG, medical and neurobehavioral performance evaluation, and subjective alertness evaluations while awake. In addition, throughout the intern's prior elective and ICU/CCU rotations, they will be subject to continuous activity monitoring and will keep a daily sleep/nap/work schedule diary. Outcome measures include adverse events (errors in procedures, diagnoses, and test interpretation).
Staff Nurse Fatigue and Patient Safety
Ann E. Rogers, Ph.D., R.N., F.A.A.N., associate professor at the University of Pennsylvania School of Nursing, described an ongoing study to determine if there is an association between the hours worked by staff nurses and errors. (A second phase of the study will focus on interventions designed to reduce errors.) The study is evaluating a randomly selected national sample of 329 full-time hospital staff nurses, using logbooks to collect data for 28 days regarding sleep patterns, scheduled work hours, actual work hours, overtime, errors, and near misses.
Findings on Work Hours
The majority of hospital staff nurses no longer work traditional eight-hour day, evening, and night shifts. In fact, 58.6 percent of scheduled shifts are 12 hours, compared to only 38.3 percent scheduled for 8 hours. All participants reported that they worked more than their regularly scheduled hours and that they worked more than 40 hours per week at least once during the 28-day data collection period. Some nurses worked more than 40 hours during all four weeks of the study, with the highest number of hours worked in a week being 117, a figure that rivals the work week of residents. Most nurses get few real breaks during their shifts.
These long working hours may be difficult for nurses, who tend to be older and have family responsibilities. The median age of the respondents is 45. Many have children, while 15 percent regularly care for an elderly parent. Some nurses care for both children and an older individual.
Findings on Errors and Near Misses
Just under one third (31 percent) of respondents reported making an error during the data-gathering period, while slightly more (33 percent) reported a near miss. Medication-related events are the most frequent type of mishap, responsible for just under 60 percent of both errors and near misses. Delivering the medicine at the wrong time was the most frequent type of medication error (accounting for 33.6 percent of all medication errors), followed by incorrect dose (24.1 percent), incorrect drug (17.2 percent), and omission (15.5 percent). Almost giving the wrong drug was the most frequent type of medication-related near miss (responsible for 29.3 percent of all near misses), followed by almost giving the wrong dose (21.6 percent), almost giving the drug to the wrong patient (19.0 percent), and almost forgetting to give the drug (16.4 percent).
The next study tasks involve analyzing the relationship between staff nurse work hours and errors. The study is also being replicated with another randomly selected national sample of full-time ICU staff nurses.
AHRQ Web M&M: Building an Online Patient Safety Journal
Robert E. Wachter, M.D., professor of medicine and epidemiology at the University of California at San Francisco (UCSF), associate chairman of UCSF's Department of Medicine, and chief of the medical service at UCSF Medical Center, reviewed AHRQ's initiative to create an online Morbidity & Mortality (M&M) Conference.
In issuing the RFA to develop this online tool, AHRQ called for the initiative to be organized around five clinical specialties (medicine, surgery, ob/gyn, pediatrics, and psychiatry), targeting the chief resident in each specialty. It also called for the creation of an editorial board consisting of clinical experts, safety experts, and trainees. The original idea was to use anonymous case reporting of near misses (to avoid liability concerns about reporting on actual errors). Dr. Wachter believes that a spectacular team has been assembled to launch this product, with UCSF editors playing a key role, along with Web developers from DoctorQuality, Inc.™ (an independent company), an editorial board, an advisory panel of more than 60 experts in clinical disciplines and patient safety, and several key staff at AHRQ.
The project has evolved since the RFA was released. The "near-miss" rule for selecting cases has evolved to the "no-permanent-harm" rule—that is, a case can be used as long as the victim of the error reached his or her "pre-error state." This rule may be relaxed in the future. The five designated clinical areas were reduced to four (psychiatry was dropped), with the fifth slot being designed a "wild card" to allow occasional inclusion of cases from other specialties. In addition, a "find another specialty" function was added to allow users to find cases that may have relevance to non-designated specialties.
AHRQ has engaged in a significant marketing effort along with other activities to improve its appeal to users. The site is designed to be visually appealing and lively, and offers opportunities for continuing medical education (CME) credits and trainee certification for residents to meet ACGME (American College of Graduate Medical Education) requirements related to patient safety education and systems thinking. To ensure anonymity and confidentiality while offering CME, AHRQ has developed a separate CME registration process.
Dr. Wachter provided a quick tour of the site, showing a case from the first issue, released February 3, 2003. (New issues will be released each month). Within each case write-up, users can click to see references, charts and figures, greater detail and commentary, and take-home points. They can also access comment forms and download slide shows.
User statistics from the first three weeks after the launch (February 3 through February 26, 2003) show substantial interest in the online journal. Promotional activities appear to be effective in boosting interest, as cumulative user registrations have jumped after the release of electronic newsletters and advertisements promoting the journal.
Dr. Wachter believes that the experiences with the online M&M suggest that there is a strong demand for patient safety education that is case-based, lively, accessible, and interactive. Providers are eager to report cases if it is safe to do so, and such cases represent an important piece of learning. In addition, the Web is an ideal vehicle for leveraging the value of such educational tools. Future plans include use of videos, quizzes, and other tools, and the possible development of a new portal for other users (e.g., pharmacists, ambulatory care) and new dissemination vehicles (there is interest in translating the site into Japanese).
Furthering the User-Driven Patient Safety Research Agenda
The third panel focused on strategies for furthering a patient safety research agenda that is driven by the users of research, including hospitals, physicians, State and Federal agencies, and patients. Nancy Foster, senior associate director for the AHA and facilitator of the panel, emphasized the importance of involving the users of research in the planning and development of that research.
The State Policy Maker Perspective
Mark Gibson, program officer at the Milbank Memorial Fund and former policy advisor for health and human services for the Office of the Governor, Oregon, offered the perspective of a State policy maker on the challenges in putting research results to use. His advice to researchers is to work closely with State policymakers in an effort to maximize the impact from research.
Understanding the Terrain
Mr. Gibson began with an overview of the State legislative world facing researchers as they attempt to work with policymakers. Within the executive branch, researchers must seek the support of both career service officials and political appointees. Career service officials tend to be detail oriented and, especially in large States, may have substantive knowledge on health care issues, including patient safety. While their resistance can slow progress, they tend to have time to spend on issues and therefore can be important allies. Political appointees, on the other hand, are less detail oriented. Nonetheless, issues of patient safety should resonate with them. The extensive bureaucracies of large States may slow progress, but these States are also more likely to have personnel with familiarity and interest in patient safety issues.
Within the legislative branch, the knowledge and interests of individual legislators will vary tremendously. But one key legislator can often drive an agenda. Legislative staffs can also be important allies. Some staff are political appointees with high rates of turnover; these individuals should not be overlooked because they may have the trust and loyalty of the legislator. There may also be some career legislative staff who can be valuable advocates for patient safety. Because some legislatures convene for only three to six months each year, patient safety advocates may have to visit key legislators in their home towns when the legislature is not in session.
Challenges in Using Research Information
Members of both the executive and legislative branches of State government face significant—but not insurmountable—barriers in using research. For many State officials, the applicability of such research may not be immediately apparent. Researchers need to find ways to make their own agendas relevant to the agendas of these individuals. Legislators, staff, and members of the executive branch also may find it difficult to understand patient safety research. Mr. Gibson encouraged researchers to develop general, brief summaries of key points, letting the audience ask for more details if they so desire. Most important, perhaps, State policymakers may find it difficult to understand how to translate the research into useful policy. Mr. Gibson urged researchers to carefully consider whether their findings have public policy implications before approaching State officials. In some cases, research findings might be more relevant to private organizations, such as hospitals or physician groups.
Even when findings are relevant to State policy, most States are facing severe resource constraints, both in terms of analytical capacity to evaluate the wisdom of new programs and, more importantly, funding for such programs. While savvy researchers can assist with the analytical piece, the funding issue is harder to overcome. Programs that will save money for States will naturally be better received, especially if the upfront costs are low. Finally, even policy changes where the benefits are clear will face sophisticated opposition. Mr. Gibson urged researchers to think carefully about what groups may support or oppose change, and to cultivate potentially powerful allies to serve as a countervailing force to the opposition.
The Federal Government Perspective
Shirlie E. Kellie, M.D., M.Sc., medical officer in the Division of Clinical Standards and Quality in the Kansas City Regional Office of CMS, offered the perspective of a Federal official on challenges in putting patient safety research to use.
Tapping into the QIO Network
Dr. Kellie focused on CMS's quality improvement organizations or QIOs, a network of 50 organizations that work to improve the health and health care of Medicare beneficiaries in each State. Like the agricultural extension service, QIOs can be quite useful to researchers. Research findings feed into CMS-QIO developmental projects that focus on a set of national priority areas established by CMS (e.g., heart failure, diabetes). QIOs, in turn, work with hospitals, nursing homes, home health agencies, and ambulatory care facilities on projects oriented at improving care in these priority areas.
Dr. Kellie emphasized the importance of not making knowledge transfer a "one-way street." Noting that published articles alone often do not promote behavior change, she advocates in-person meetings as a vehicle for knowledge transfer. For example, in late February 2003, she took part in a meeting between developers of the Medical Events Reporting System (MERS) and potential users of the system. This approach allowed the researchers who developed the system to gain valuable user insights that can be used to modify MERS so that it is more likely to have a positive impact on patient safety. To further this effort, CMS and the QIOs are launching a pilot study in which four QIOs will each work with two hospitals on implementing MERS.
The Hospital and Health System Perspective
Gina Pugliese, R.N., M.S., vice president of the Premier Safety Institute and an associate faculty member at the University of Illinois School of Public Health (Division of Epidemiology and Biostatistics) and the Rush University College of Nursing, provided the hospital and health system perspective on the challenges in using patient safety research. The Premier Safety Institute is part of Premier, Inc., a strategic alliance of 1,700 not-for-profit hospitals in the US.
The biggest challenge facing hospitals and health systems relates to coordinating research findings for those who need to use them. Hospitals and health systems face the following:
- Information overload, including 400,000 new research articles each year, 1,500 new drugs coming on line over the next three years, and millions of health care Web sites. Hospitals implementing patient safety practices must sort through tremendous amounts of information.
- Multiple guidelines, practices, and performance measures, making it hard to know who is recommending what based on the evidence.
- Difficulty in making comparisons, due to conflicts and inconsistencies across recommendations, in part due to a lack of common vocabulary.
Overcoming These Challenges
Ms. Pugliese noted that there are many vehicles to help clinicians gain access to and make sense of the tremendous amounts of information available, including the AHRQ Web site, general and trade press, journals, professional organizations and Web sites, Federal agencies, librarians, and schools. Many of these sources are quiet helpful and useful, but they also represent "all the usual suspects" in terms of dissemination. She called for more creative thinking about how to make users of research more likely to actually implement change. One important strategy for achieving this goal is the development of just-in-time information through searchable databases that cut across organizations and Web sites. Ms. Pugliese called for increased investments in tools that synthesize advice, such as AHRQ's https://talkingquality.ahrq.gov Web site that provides information on how to present quality data to consumers.
Ms. Pugliese laid out a series of questions for researchers as they publish. The answers to these questions will help health care practitioners to implement research findings.
- Is there sufficient evidence of the effectiveness of the intervention?
- Is it more effective than alternatives with respect to the same safety goal?
- What are the barriers to implementation?
- Is there evidence of successful transfer to non-research settings?
- Can information be targeted to different settings?
- Are there tools for measuring the effectiveness in terms of performance and process?
- Will implementation of the practice have a positive or negative impact on revenues?
- Are there potential unintended consequences?
- Is there an estimate of the resources required to implement the research practice in question? What are the minimum staffing requirements?
In answering these questions, researchers should be honest about what is and is not known, and should try to provide potential users with assistance in prioritizing interventions to implement. AHRQ's Making Health Care Safer: An Evidence-Based Compendium is a valuable (but little known) resource to help potential users with this type of prioritization.
A study of organizations participating in IHI (Institute for Healthcare Improvement) Collaboratives suggests that successful interventions are not dependent upon hospital characteristics (e.g., size, location, teaching status, ownership), but rather on the presence of strong leadership (both the CEO and team leaders) and effective processes, including having clear aims, being able to quantify progress, forming and using interdisciplinary teams, securing the early involvement of key stakeholders, focusing on practical interventions that emphasize changing processes (not people), and developing easy-to-collect and measure outcomes data. (Not surprisingly, the biggest barriers to success occur when these characteristics are not in place.)
The Clinician Perspective
Gordon Schiff, M.D., senior attending physician and director of clinical quality research in the Department of Medicine at Chicago's Cook County Hospital, offered the clinician's perspectives on the challenges in putting research results to use.
He began by highlighting the many stresses that practitioners face that multiply the opportunity for errors, including the increasing complexity of care systems and patients, drug toxicity and complexity, a raised bar with respect to the expectations placed on physicians (e.g., related to disease management), staffing stresses (e.g., shortages), discontinuities and care fragmentation, shift handoffs, overcrowded units and emergency departments, quicker admissions and sicker patients, documentation requirements, information overload, billing and administrative inefficiencies, and financial barriers.
When practitioners face multiple stresses, the opportunity for error increases significantly. At Cook County Hospital, for example, patients must wait in lines that extend for two blocks to see a pharmacist. As a result, pharmacists are rushed and have little opportunity to counsel patients. Dr. Schiff is particularly concerned about several seldom mentioned medication issues, including errors made when modifying drug regimens (most computerized systems do not have information on existing regimens—i.e., patient drug history and experience), identifying tablets, capturing reasons for the discontinuation of drugs, capturing medication indication, synchronizing medications (a confusing issue for patients with multiple chronic illnesses), and counseling.
A "Wish List" to Improve Safety
Dr. Schiff reviewed a "wish list" of actions and strategies that will help clinicians to improve patient safety. Researchers can play a crucial role in bringing these to fruition.
- Make it easier and safer for clinicians to learn from mistakes.
- Help tie reporting to feedback and actual improvement.
- Help figure out how to motivate practitioners to welcome external review and regulations.
- Help overcome interruptions, distractions, and over-reliance on memory.
- Help integrate routine care with research collaboration.
The Patient Perspective
Sorrel King is the mother of Josie King, who died in 2001 at 18 months of age at Johns Hopkins University Medical Center as the result of system-related hospital errors. These errors occurred despite the fact that Ms. King repeatedly questioned caregivers about her daughter's signs of dehydration and the hospital's use of narcotics. At one point doctors had ordered that the narcotics no longer be administered, but a nurse who was not aware of the order administered the drug anyway. Shortly afterward, Josie King died. In response to this tragedy, Ms. King and her husband created the Josie King Pediatric Patient Safety Program at the hospital's children's center. This program identifies safety concerns, revised medical education to sharpen the focus on proactive safety measures, and empowers families to become equal partners in their child's care.
Ms. King highlighted several key lessons that she learned during her tragic experience:
- Efforts to improve patient safety must address the hearts, minds, and souls of caregivers, who must learn that they are fallible. Patients and families must be treated as true partners, their concerns and requests taken seriously. Without this type of culture change in medicine, other attempts to improve safety will likely be ineffective.
- Nursing and medical school education must change. Students should hear about the IOM report and should see the faces behind the chilling statistics on deaths from medical errors, a subject that should be treated as an epidemic.
- The system must move towards disclosure, openness, and transparency. Family members and patients who are victims of errors are not primarily looking for money, but they want the health care system to tell the truth, to sincerely apologize, and to fix the problems so that no one else falls victim to the same mistakes in the future.