The Agency for Healthcare Research and Quality (AHRQ) sponsored a theme issue of the journal Health Services Research: April 2009 44 (2), Part II. Eight new studies in this issue, "Program Evaluation of the AHRQ Patient Safety Initiative," take an in-depth look at AHRQ's own patient safety initiative, including the findings from a 4-year RAND Corporation evaluation of more than 300 research projects and other activities. In the first paper, RAND researcher Donna Farley, Ph.D., and AHRQ researcher James Battles, Ph.D., describe the purpose for this supplemental issue, as well as the framework and approach to evaluating these patient safety initiatives.
Six process evaluation papers take a closer look at the assessment and experiences of AHRQ-funded patient safety projects, information technology initiatives, the AHRQ-sponsored Patient Safety Improvement Corps, and the growth of patient safety partnerships across the United States. The final two papers explore the challenges inherent in measuring safety outcomes and present the overall findings from the 4-year evaluation. Brief summaries of the papers follow.
Farley, D. O., and Battles, J. B., "Evaluation of the AHRQ patient safety initiative: Framework and approach," pp. 628-645.
Serving as the Patient Safety Evaluation Center, the RAND Corporation conducted a 4-year evaluation of AHRQ's initiative dealing with diverse patient safety issues and practices. This lead paper describes the Institute of Medicine's (IOM) mandate for this initiative and its evolution to date, as well as the program evaluation methods used to evaluate it. The authors explain the specific components of the Context-Input-Process-Product (CIPP) evaluation model selected by AHRQ, its application to the initiative, and the process of data collection and analysis. As the patient safety initiative has matured over the years, it has moved from basic knowledge development to testing actual practices deemed effective and informing end users about these outcomes.
Sorbero, M. E. S., Ricci, K. A., Lovejoy, S., and others, "Assessment of contributions to patient safety knowledge by the Agency for Healthcare Research and Quality-funded patient safety projects," pp. 646-664.
AHRQ's extensive portfolio of projects has addressed a wide range of patient safety issues, not only in hospitals, but also in other healthcare environments. Most of the projects focus on general patient safety and medication ordering or administration issues. Special populations, such as the elderly and minorities, are adequately included in the portfolio. Projects most often focus on such patient safety actions as the monitoring and reporting of adverse drug events, provider education and awareness, physical environment redesign, and technology innovation to reduce errors. Many of these patient safety projects look at practices that still need more scientific evidence before their effectiveness can be completely determined. The RAND team recommends that AHRQ develop and implement a strategy to make sure project-generated knowledge and results reach front-line health care providers through organizational collaboration and the creation of new learning tools.
Taylor, S. L., Ridgely, M. S., Greenberg, M. D., and others, "Experiences of Agency for Healthcare Research and Quality-funded projects that implemented practices for safer patient care," pp. 665-683.
In this study, RAND researchers interviewed 60 groups who received 1 of 3 types of patient safety grants (original, challenge, and partnership) from AHRQ between 2003 and 2006. The self-reported data point to a number of similarities in shared experiences of the grantees, particularly when it comes to lessons learned. Key components of successful project implementation include existing partnering organizations, narrowing the project scope, the use of "champion" buy-in, and technical assistance availability. There were also several unexpected challenges, such as time delays, physician buy-in, and the actual difficulty in changing staff practices. This study represents one of the first reported evaluations of patient safety implementation projects in the literature.
Damberg, C. L., Ridgely, M. S., Shaw, R., and others, "Adopting information technology to drive improvements in patient safety: Lessons from the Agency for Healthcare Research and Quality health information technology grantees," pp. 684-700.
Electronic health records (EHR), computerized physician order entry (CPOE), and data exchange networks are just some of the ways information technology (IT) is being used to improve patient safety in health care settings. AHRQ has promoted the innovative use of IT in this area by awarding grants totaling $139 million to 104 health IT projects in 2004. This paper looks at the experiences of these projects and the lessons learned from their adoption, implementation, and long-term viability. Overall, the projects represented diverse facilities, geographic locations, and special populations served. The projects use IT to influence decision support and to implement EHR and CPOE. Important factors linked to the success of implementing patient safety IT projects were identified. These include end-user engagement, adequate pilot testing, effective communication, good leadership, and available training opportunities. The majority of grantees felt strongly that their gains in knowledge and implemented systems could be successfully replicated by other institutions.
Teleki, S. S., Damberg, C. L., Sorbero, M. E. S., and others, "Training a patient safety work force: The patient safety improvement corps," pp. 701-716.
As part of its patient safety initiative, AHRQ created the Patient Safety Improvement Corps (PSIC). This 1-year program trained patient safety teams in all 50 States and the District of Columbia for 3 years between 2003 and 2006. This study by the RAND team evaluated how yearly training assisted in creating a national infrastructure to support patient safety practices. After 1 year post-training, participants reported coming away with valuable skills and the confidence to use them in their health care settings. They continued to use these effective patient safety interventions even after 2 years had passed since receiving the training. Its impact reached well beyond the immediate health care environment to affect patient safety actions made by state agencies in the form of legislation and oversight procedures. Following their training, participants became effective trainers of others at their institutions. Despite training approximately 250 individuals, the PSIC needs to strengthen and expand this network in order to create long-term improvements in patient safety practices and outcomes. Continued training of new participants as well as postgraduate modules for former trainees will help.
Mendel, P., Damberg C. L., Sorbero, M. E. S., and others, "The growth of partnerships to support patient safety practice adoption," pp. 717-738.
Part of RAND's evaluation of AHRQ's patient safety initiative also included a critical look at the formation of organizational partnerships at the national level. In this study, researchers conducted two rounds of telephone interviews with 35 organizations in 2004 and 55 in 2006. These key organizations were found to be engaged in interorganizational partnerships of various sizes and reach. Over the time period, there was a considerable increase in notable activities, particularly the dissemination of patient safety information and the development of practical tools. Partnership network fragmentation decreased while network centralization increased. The researchers found that AHRQ was centrally positioned within these partnerships to be a leader in these activities. As patient safety partnerships of all types continue to grow and expand, AHRQ's central role in these collaborations ensures the ongoing creation and dissemination of tools and practices.
Greenberg, M. D., Haviland, A. M., Yu, H., and Farley, D. O., "Safety outcomes in the United States: Trends and challenges in measurement," pp. 739-755.
Patient safety is a complex issue that interrelates with a variety of health care settings, therapies, and patient risk levels. Such intricacies make tracking national trends difficult and challenging. Such is the conclusion of RAND researchers who analyzed data from a variety of medication error and other event-related databases. Each of these demonstrates the benefits and limitations of safety reporting systems. For example, the Joint Commission Sentinel Events database identifies the various factors causing serious health care incidents. However, incomplete reporting difficulties result in underestimates. This illustrates the limitations found in other external reporting systems, particularly those that are voluntary. These inherent limitations among databases negatively impact the ability to create an accurate, national picture of patient safety outcomes. Based on their analysis, the RAND team recommends that AHRQ track trends, not only in patient outcome measures, but also in the way evidence-based safe practices are implemented nationally. Other suggestions include gathering and analyzing data from nonhospital health care settings, using expert consensus methods, and paying careful attention to consistent definitions and coding techniques.
Farley, D. O., and Damberg, C. L., "Evaluation of the AHRQ patient safety initiative: Synthesis of findings," pp. 756-776.
In the concluding article, overall findings are summarized from the 4-year RAND evaluation. The researchers conducted interviews with AHRQ staff, its grantees, and other patient safety parties. They also reviewed materials published by AHRQ and various internal documents. During the period from 2001 through 2006, AHRQ made strong progress in developing new knowledge on patient safety epidemiology and practices. There was also a strengthening of infrastructure in order to support the adoption of safe practices among health care providers at the institutional level. Only limited progress was made in AHRQ's ability to create a monitoring and vigilance capability. Divergent views among stakeholders and lack of consensus have hampered the development of a national monitoring capability. According to those interviewed, the lack of engaged health care leadership and continued denial of safety issues within the health care community are just some of the reasons why progress remains frustratingly slow. Along with being grateful to AHRQ for its work to date on patient safety despite limited resources, stakeholders suggest that it work harder at getting evidence-based practices adopted by front-line providers. Partnering with other organizations may help with efficient and timely dissemination.
Editor's Note: A limited number of copies of the HSR theme issue (AHRQ Publication No. OM09-0059) are available from the AHRQ Publications Clearinghouse.