Chapter 7. Trending: Comparing Results Over Time
2010 User Comparative Database
Many hospitals that have administered the hospital survey have indicated that they intend to readminister the survey on a regular basis to track changes in patient safety culture over time. Some of the hospitals that previously administered the survey and submitted data to the database then readministered the survey and submitted data again. The overall results presented earlier in this report reflect only the most recent survey data from all 885 participating hospitals. But we have data from two or more administrations of the survey for 321 hospitals, allowing us to examine trends over time for these hospitals. This chapter presents trending results from these 321 hospitals. Changes of 5 percentage points or more are highlighted.
When reviewing the results in this chapter, keep in mind that the trending results from these 321 hospitals represent approximately one-third of the total number of database hospitals. Therefore, the trending data should be viewed as preliminary. In addition, survey scores might change, or not change, over time for a number of complex reasons. Important factors to consider are whether the hospital implemented patient safety initiatives or took actions between survey administrations and the length of time between administrations.
Survey methodology issues can also play a big role in score changes. It can be difficult to interpret changes in scores over time for a number of reasons. These include low survey response rates for the previous or most recent administration, changes in the number of staff asked to complete the survey, and changes in the types of staff asked to complete the survey.
Table 7-1 displays summary statistics from the previous and most recent survey administrations for the 321 trending hospitals. As shown in the table, the average number of completed surveys increased in the most recent survey administration (from an average of 357 to 399 respondents). The average response rate also increased (from 49 percent to 56 percent).
Additional characteristics of the 321 trending hospitals follow:
- Most of the 321 trending hospitals (71 percent) administered the survey to the same types of staff in their previous and most recent administrations.
- The average change in response rate from the previous administration was 7 percent (range: one hospital had an 85 percent decrease in response rate and one had an 85 percent increase).
- The average time between the previous and most recent survey administrations was 19 months (range: 6 months to 46 months).
Note: Descriptive statistics of the 321 trending hospitals by bed size, teaching status, and ownership and control are provided in Appendix C (Tables C-1, C-2, and C-3).
Description of Trending
Before presenting results on the changes in survey scores over time, we provide an explanation of the trending statistics that are presented. Table 7-2a shows examples of the statistics shown in this chapter. The tables show the average percentage of respondents who answered positively in the most recent survey administration (left column) and the previous administration (middle column) for the 321 trending hospitals only. The change over time (Most Recent Score minus Previous Score) is shown in the right column. The change is a negative number if the most recent administration showed a decline and a positive number if the most recent administration showed an increase.
Table 7-2b shows additional trending statistics that are provided. The maximum increase and maximum decrease show the scores for the hospitals with the largest average percent positive score increase and the hospitals with the largest decrease. The average increase and decrease of percent positive scores across the 321 trending hospitals is also shown. The average increase was calculated by only including hospitals that had an increase in their most recent score; hospitals that showed no change or decreased were not included when calculating the average increase. Similarly, the average decrease was calculated by only including hospitals that had a decrease in their most recent score; hospitals that showed no change or increased were not included when calculating the average decrease.
Composite and Item-Level Trending Results
Table 7-3 presents trending results showing average percent positive scores on each of the 12 patient safety culture composites from the 321 trending hospitals. The table shows percent positive scores for the hospitals' most recent and previous data administration/submission. The table also shows the average change over time, the hospital scores with the maximum increase and maximum decrease, and the average increase and decrease over time across the 321 hospitals.
Table 7-3 also shows a slight overall increase in the average change in percent positive scores over time on the patient safety culture composites (average 2 percentage points, ranging from 1 to 3 percentage points). For hospitals with increases in scores over time, average increases ranged from 5 to 7 percentage points. For hospitals with decreases in scores, average decreases ranged from 4 to 6 percentage points.
The item-level trending results in Table 7-4 show that the average change in item-level percent positive scores over time on the patient safety culture items ranged from an increase of 1 percentage point to 4 percentage points. For hospitals with increases in item scores over time, average increases ranged from 6 to 10 percentage points. For hospitals with decreases in item scores, average decreases ranged from 4 to 8 percentage points.
Trending results from the item that asks respondents to give their hospital work area/unit an overall grade on patient safety are shown in Table 7-5. The average percentage of respondents giving their work area/unit a patient safety grade of "A-Excellent" and "B-Very Good" increased over time by 4 percentage points.
Trending results from the item that asked respondents to indicate the number of events they had reported over the past 12 months are shown in Table 7-6. The average percentage of respondents reporting one or more events decreased slightly over time by 1 percentage point.
Bar Charts of Trending Results
The bar charts in Chart 7-1 show the percentage of hospitals that increased, decreased, or did not change by 5 percentage points or more for each of the 12 patient safety culture composites. These charts show that:
- Most hospitals changed less than 5 percentage points on the 12 composites (ranging from 46 percent to 63 percent of hospitals on each of the composites).
- The composites with the largest percentage of hospitals that increased 5 percentage points or more were Management Support for Patient Safety and Staffing (38 percent of trending hospitals increased by at least 5 percentage points).
- The composite with the largest percentage of hospitals that decreased 5 percentage points or more was Handoffs and Transitions (23 percent of trending hospitals decreased by at least 5 percentage points).
Chart 7-2 displays trending results for the percentage of respondents providing patient safety grades of "A-Excellent" or "B-Very Good" and shows that:
- 41 percent of hospitals increased by 5 percentage points or more.
- 40 percent of hospitals changed less than 5 percentage points.
- 19 percent of hospitals decreased by 5 percentage points or more.
Chart 7-3 displays trending results for the percentage of respondents reporting one or more events and shows that:
- 23 percent of hospitals increased by 5 percentage points or more.
- 48 percent of hospitals changed less than 5 percentage points.
- 29 percent of hospitals decreased by 5 percentage points or more.
Chart 7-4 displays the number of composites on which hospitals increased, decreased, or did not change:
- Most hospitals (77 percent) increased by 5 percentage points or more on at least one composite.
- About half of the hospitals (51%) decreased by 5 percentage points or more on at least one composite.
- About half of the hospitals (48%) changed less than 5 percentage points on 7 or more composites.
Additional Trending Analyses
The following sections present quantitative and qualitative data on changes in patient safety culture over time. The quantitative data include questionnaire data on actions taken by the trending hospitals to improve their patient safety culture. The qualitative data consist of findings from nine interviews conducted with staff at trending hospitals and suggest explanations for increases and decreases in hospitals' hospital survey scores.
About 91 percent (292) of the 321 trending hospitals (hospitals that administered the patient safety culture survey and submitted data more than once) provided basic information about the types of patient safety actions they had taken in between their previous and most recent survey administrations.
Table 7-7 shows the percentages of trending hospitals that shared their previous survey results with various groups of people. Most of the trending hospitals (94 percent) that provided such information reported that they had shared their previous survey results with hospital administrators. In addition, 83 percent reported they had shared their previous survey results with department managers, and 72 percent reported they had shared their previous survey results with hospital staff. Fewer hospitals reported they had shared the results with physicians (53 percent) or their Board of Directors (52 percent). Nine hospitals (3 percent) reported that they had not shared their previous survey results yet.
Table 7-8 shows the percentages of trending hospitals that reported they had implemented various types of actions. The action most frequently taken was implementing the Situation-Background-Assessment-Recommendation (SBAR) technique (65 percent).
Most of the trending hospitals (92 percent) implemented more than one action.
To gain a better understanding of changes in patient safety culture and patient care practices over time, hour-long telephone interviews were conducted with staff from nine hospitals that administered the hospital survey more than once. The interviews were conducted in fall 2008. Six of the hospitals experienced notable increases in their scores, and three hospitals experienced notable decreases. Most interview participants were quality/risk managers, and one was a chief executive officer. The nine hospitals varied with respect to system affiliation, bed size, teaching status, ownership, and geographic region.
Explanations for notable increases in hospital survey scores. During the interviews, participants were asked why their hospitals' survey scores increased. Some participants mentioned specific actions, including:
- Implementing the SBAR communication tool for unit-to-unit transfers.
- Hiring a consultant group to work with department directors on targeted patient safety problems.
- Addressing staffing requirements such as filling nursing vacancies and improving patient/staff ratios, and
- Using and displaying scorecards to monitor progress on hospital initiatives.
Generally, various themes emerged from their responses. These themes are shared here, along with participants' comments about actions their hospitals took to improve patient safety culture and safe patient care practices. Four main themes emerged from those hospitals with notable increases in their hospital survey scores.
Theme 1: Hospitals improved their communication between management and staff on patient safety
Sample Actions and Illustrative Quotes
- Conducted walkarounds to learn about staff concerns about patient safety.
- Focused on patient safety during staff meetings.
- One participant attributed her hospital's improvement to "the engagement of our department heads and nursing coordinators in making sure patient safety culture is on everyone's mind."
- Started conducting monthly staff meetings.
- Implemented Open Book Management and participated in biweekly "huddles" to review the hospital budget, financial statements, and patient safety issues and concerns.
- "Open Book Management has had the biggest impact of all their initiatives.affected everything we do.employees are much more aware."
Theme 2: Hospitals focused on improving error reporting systems, responding appropriately to reports, and applying nonpunitive "Just Culture" principles
- Educated hospital leaders on making error reporting anonymous, easy, and convenient.
- "When we went from a paper system to an electronic system, our reporting increased about 40 percent—part of it was education, because we had to do a lot of education as we rolled out the electronic system—part of it.is because it's very easy."
- Set up a hotline for reporting errors and developed anonymous reporting forms for medical errors.
- "We got management to buy into that it was o.k. for a staff person to not provide their name, so they wouldn't be afraid to report."
- Trained staff to use the new reporting systems.
- Provided training on "Just Culture" and taught managers to use an algorithm when examining patient safety error incidents.
- "The algorithm helps management more than anything else."
Theme 3: Hospitals engaged staff in developing solutions to patient safety problems
Sample Actions and Illustrative Quotes
- Directly involved staff in designing solutions to handoff problems.
- Started an employee engagement committee that includes senior leaders.
- Instituted nursing peer review to promote open communication.
- "I personally think it is a combination of the employee engagement committee where employees have a voice. I think it's the peer review. Having peers to go to, to voice your concerns."
- Assigned staff to a scheduling team to accommodate staff preferences.
- Allocated resources for safety needs identified by staff—for example, buying safer beds.
Theme 4: Hospitals developed, implemented, and monitored action plans, in some cases focusing on specific survey items
- Charged department managers with developing and implementing an annual action plan and held them accountable.
Explanations for notable decreases in hospital survey scores. Interview participants provided the following explanations as possible reasons for decreases in their scores in their most recent administration of the survey. Explanations for decreases in scores tended to be quite specific and unique to the hospital being probed:
- Experienced issues among staff with specific managers and management styles, especially regarding managers' response to incident reports and lack of followup on staff feedback.
- "They felt like the managers really didn't act on them [incident reports] or hear them or do anything about them."
- Had contracting issues and high turnover for managers and frontline staff—staff have had to get used to new unit managers; some new managers not familiar with hospital policies on "Just Culture"
- Needed to temporarily shut down hospital services because contract and financial constraints led to a large shortage of professional providers.
- "The staffing issue came up as part of contract problems. We're in a fairly isolated area, and we have a vacancy rate in the professional provider staff of about 40%. During this timeframe, we also changed financial management systems. We're not able to hire contractors with the speed that we had in the past. We ended up running very short and ended up closing beds and shutting services down for about an 18-month period."
- Drilled down in the survey data and observed that scores were lower for larger than smaller units—attributed the lower scores to less frequent and personal communications, weaker sense of accountability to coworkers.
- Were in the middle of union negotiations and staff were feeling hostile.
- Struggled with organizational learning and how much information could be fed back to staff given confidentiality requirements and concerns.
- "As we run into significant adverse events for patients, how much do we feed the information back to frontline staff? Where's that line of keeping it confidential yet sharing our learnings with staff?"
Appendixes C and D: Trending Results by Hospital and Respondent Characteristics
Part III of the report contains Appendixes C and D that show trends over time for the 321 hospitals that administered the survey and submitted data more than once. Average percent positive scores from the most recent and previous administrations are shown on the survey composites and items, broken down by the following hospital and respondent characteristics:
Appendix C: Trending Results by Hospital Characteristics
- Bed size.
- Teaching status.
- Ownership and control.
Appendix D: Trending Results by Respondent Characteristics
- Work area/unit.
- Staff position.
- Interaction with patients.
Because there are many breakout tables, they are included in Appendixes C and D. Highlights of the findings from the breakout tables in these appendixes are provided on the following pages. The appendixes are available on the Web at: http://www.ahrq.gov/professionals/quality-patient-safety/patientsafetyculture/hospital/2010/index.html.
Note 1: Because there were fewer than 20 trending hospitals in several hospital region breakout categories, trending results are not shown by hospital region, to ensure hospital confidentiality.
Note 2: New to the 2010 database, breakouts by respondent characteristics (Appendix D) were only calculated for hospitals that had at least five respondents in the breakout category. If a hospital had fewer than five respondents in a certain category, the hospital is not included in the statistics displayed for that category. (Further explanation is in Notes: Description of Data Cleaning and Calculations.)
Highlights From Appendix C: Trending Results by Hospital Characteristics
Bed Size (Tables C-4, C-6, C-7)
- Large hospitals (400-499 beds) had the greatest increases in percent positive response over time on 7 of the 12 composites (average increase of 5 percentage points across these 7 composites).
- Small hospitals (6-24 beds) had the greatest increase in percentage of respondents who gave their work area/unit a patient safety grade of "Excellent" or "Very Good" (a 7 percentage point increase, from 73 percent in the previous administration to 80 percent in the most recent administration).
Teaching Status and Ownership and Control (Tables C-8, C-10, C-11)
- There were no noticeable changes over time on the patient safety culture composites by teaching status or ownership and control (all changes were 4 percentage points or less).
Highlights From Appendix D: Trending Results by Respondent Characteristics
Work Area/Unit (Tables D-1, D-3, D-4)
- Obstetrics had the greatest increase in percent positive response on 5 of the 12 patient safety culture composites (average increase of 6 percentage points across these 5 composites).
- ICU, Pediatrics, and Pharmacy shared the greatest increase over time in average percentage of respondents giving their work area/unit a patient safety grade of "Excellent" or "Very Good" (each increased by 6 percentage points).
- There were no noticeable increases in the average percentage of respondents reporting one or more events in the past year. The largest decrease was in Anesthesiology (a 10 percentage point decrease).
Staff Position (Tables D-5, D-7, D-8)
- Administration/Management had the greatest increase in positive response over time on 7 of the 12 patient safety culture composites (average increase across the 7 composites was 5 percentage points).
- Therapists had the largest increase over time in average percentage of respondents giving their work area/unit a patient safety grade of "Excellent" or "Very Good" (5 percentage point increase).
Interaction With Patients (Tables D-9, D-11, D-12)
- There were no noticeable changes over time in the patient safety culture composites by level of interaction with patients (all changes were 4 percentage points or less).