Be Prepared: Lessons from an Extended Outage of a Hospital’s EHR System

Another hurdle to simply dusting off old paper records is that they quickly become out of date. Old paper records that have been sitting on a shelf are static documents that have not kept up with the evolution of actual processes at the hospital, which means they quickly become a poor substitute for electronic software. Old paper forms will likely require a thorough review and updating in order to make sure they are usable. Reviewing and updating during an outage will cause you to lose valuable time. As a best practice, we have adopted a policy of doing the review and update proactively on a concurrent basis to ensure that paper medical records can immediately be used in the case of an emergency.

We should also mention that our staff came up with a novel solution for makeshift paper medical records that worked for a while before we recognized its limitations. Once we realized that our EHR system would not be brought back online immediately, our clinical staff began using a printed version of the online interface of our EHR software. It was exactly what they would see on the screen, but it was printed on paper. That had the advantage of being familiar, was in a format that would make data entry simple once the EHR system was restored, and reflected the current processes and workflow built into the EHR. It was a clever solution and it worked for a day or so before its limitations became clear: it does not take long for this style of medical record to grow to 50+ pages long, making them monstrous documents that are difficult to work with. Each time the nursing unit got new lab results or a new progress note from a doctor, it meant another piece of paper to put in the chart. The pages grew quickly, becoming unwieldy and confusing to the staff. The answer was to have true “traditional” paper records, which we implemented in time to avoid any additional confusion.

As follow up to our outage, one of the critical updates we are making to our contingency plans is an organizational commitment to have continuously updated paper records that can be used at a moment’s notice. We have also made a commitment to regularly train staff, particularly new team members, to ensure that they are familiar with our paper system.

Get Ready to Put on a New Hat

By design, hospitals are regimented work environments with clear roles and responsibilities for both clinical and non-clinical staff, which ensure patient safety and allow us to efficiently manage the minute-to-minute operations of a large, complex health care organization. Everyone knows their role and how they are supposed to do their jobs, but an extended EHR system outage will likely push everyone into unfamiliar territory that alters or completely changes the hats they wear at the hospital. Everyone needs to be flexible and adaptable in order to make it work, and that starts at the top.

The first team to be impacted by the outage was our hospital’s IT staff (myself included, as CIO), who shifted into emergency response mode when the hospital information system went down. The team instituted its contingency plan and recognized the need to restore from a backup, but the specific nature of the outage prevented the plan from being effective. The IT team entered into an intensive problem-solving process. They quickly discovered that the most recent backup was corrupt, so the team could not perform a near-term restore. The team then had to go back to the most recent full backup, but that was also corrupt. What they discovered was that two major EHR failures had occurred:

1. The problem that caused the outage actually started several hours before the system became unavailable, and it eroded the system in a way that went undetected for more than eight hours.
2. The error was replicated in the back ups, which made several of them unusable.

The most recent successful and uncorrupted backup had been at noon on the day of the outage. Nine file servers needed to be restored back to that point in time. To make things more complicated, the hospital staff had to work through locating the eight hours of data that was no longer in the system. Per the master contingency plan, the hospital activated an incident command center to oversee the system restore efforts and to assist the rest of the hospital’s process of working without the EHR system. Having a single command center is a best practice outlined in our contingency plan, but the complexity and scope of the IT issues (involving our internal technical team, the EHR vendor’s support team, other external vendors and data center operations teams) supported our decision to split the command center into two: one focused on the technology restore process and one focused on hospital operations. There were some downsides to operating two parallel command centers but the strategic decision to have the IT team manage its own “war room” was necessary during the early stages of the outage.