Be Prepared: Lessons from an Extended Outage of a Hospital’s EHR System

1. Our outage involved something that most hospitals are probably unprepared for: a loss of data from the EHR system. EHR software is architected with numerous safeguards to protect against the loss of patient data in the event of downtime, so most health care organizations probably do not have a section of their contingency plans dedicated to that possibility. But it happened to us, and it could easily happen to your organization. That data loss added a layer of complexity to our recovery process, because we had to initiate a data recovery process in parallel to the EHR system recovery process. This involved recovering data from the memories of disparate medical devices (i.e., lab instruments, diagnostic imaging equipment, etc.) and information systems across the hospital in a sleuthing process that would have made Sherlock Holmes proud. It was a process we had to figure out on the fly. Your contingency plans can actually build in procedures that will prepare for that possibility, and then organize a rapid-response initiative if data is discovered missing. While we are on the topic of data recovery, we should note that our involvement in a Health Information Exchange (HIE) played an important role in helping us recover from this data loss. In 2011, we became the first hospital in Colorado to participate with the statewide HIE operated by the Colorado Regional Health Information Organization (CORHIO). As a result of that effort, lab tests, diagnostic imaging results and transcribed reports like operative reports, discharge summaries and inpatient progress notes were available for use in re-creating the patient record. Our HIE involvement was a significant asset in our recovery process, and it may play a similarly important role for your organization.
2. Another important lesson we learned through this process is the importance of having a pre-determined timeframe for enacting your contingency plan for a long-term outage. In our case, in the early hours and days of our downtime, it appeared several times that our system would be brought back on line. Those expectations did not become reality, and they delayed our decision to officially enact our long-term plan. If we had it to do over again, we would start the clock as soon as the outage began and we would officially implement the highest-level actions once we got to hour X, rather than assume that we would be saved by the potential fixes being worked on by the vendor and our diligent IT department. We also learned the importance of giving the IT staff adequate time to diagnosis and repair the problem before jumping to conclusions about when things would be “back on line.” It is important to have realistic expectations about the length of time it will take to fully diagnose and resolve the technical problems, and it is equally important to move ahead with implementation of the contingency plan at a predetermined time while the IT staff continues to do their job.
3. Another important lesson is about preparedness for secondary emergencies that occur during an IT outage. An outage of this length could easily coincide with another separate, unrelated crisis incident, causing a catastrophic event that most hospitals’ contingency plans would be unprepared for. Most hospitals have extensive plans for how to deal with a natural disaster or other incident that result in IT downtime; but contingency plans do not always anticipate the possibility of an unrelated event occurring while you are already struggling with a major IT issue. Many hospital contingency plans assume that everything else is static at the hospital, but what if there is a mass casualty event that occurs during the downtime? Are you prepared to manage a second emergency while you are trying to get your EHR system running and visa versa? We were lucky that no second event emerged during the 10-day period of our EHR outage, but luck is not a plan. We are now revising our contingency plans to address the simultaneous occurrence of a “traditional “emergency that also involves an IT outage.

Never Say Never Again to Paper

EHR systems help achieve many efficiencies, but during an extended IT outage you are going to appreciate a paper-based fallback system like a kid wanting cold, soothing ice cream after a tonsillectomy. It was not so long ago that our hospital was still using paper records extensively: as recently as two years ago, paper was still very prevalent in our operations, but we learned that it does not take long for paper-based procedures to become a distant memory. 

1. If your contingency plans for an IT outage simply suggest using your old paper records as a fallback plan until the computers are back up, you will be setting yourself up for a lot of headaches. The first obstacle to that plan is a very practical one: hospital staffs change fast, and it does not take long for your clinical and administrative team to have a lot of new faces who:1--Come from other hospitals that used very different paper processes, which means they do not have familiarity with your prior paper processes; and
2. Are young employees whose entire training has been on electronic systems, which means they have never actually seen a paper medical record.

In our case, we experienced both obstacles and there was a learning curve to help get everyone on the same page (literally).