Be Prepared: Lessons from an Extended Outage of a Hospital’s EHR System

August 30, 2013
| Reprints
First-hand report: How one hospital responded to a 10-day outage of its EHR
Be Prepared: Lessons from an Extended Outage of a Hospital’s EHR System

If your hospital has deployed an electronic health record (EHR) system, you probably have a contingency plan in the event of a system outage. After all, computing systems go down, and when an EHR system is not working, it affects nearly every aspect of a hospital’s operations, from patient care to admissions to finance to supply chain and more.

Having an effective response plan is critical for mitigating the impact of downtime, and your organization has likely put a tremendous amount of thought and care into its contingency plan. But your plan may have an Achilles’ heel that your organization is completely unaware of—a weakness that could leave your organization as poorly prepared as if you had no contingency plan at all. Where are the holes in your plan? Find them by asking a simple question: What is the longest hypothetical outage you have planned for?

Are you prepared for an EHR outage of an hour? Eight hours? Twenty-four hours? What about 10 days? 

We mention 10 days for a good reason. That is the length of an EHR outage that Boulder Community Hospital in Colorado experienced earlier this year. And we are not alone. We have heard and read about similar extended EHR outages at other hospitals, so this was not an isolated experience. It could happen anywhere, and when it does, it can undermine even the best contingency plan. Our plan’s “worst case scenario” for an EHR outage did not anticipate a situation where our entire hospital information system would be down for 10 days. We got through the outage thanks to a tremendous amount of hard work and ingenuity by members of our clinical team, administrative team and technology team, and we learned a lot of lessons and developed new best practices through that experience. 

The purpose of this article is to share what we learned, to allow your organization to integrate our lessons learned into your contingency plan to better prepare for a similar technology failure, if (or when) it happens.

How Long Is Long-Term?

One of the biggest lessons we learned through this experience is that our imagination was not pessimistic enough when anticipating what would constitute “long-term” downtime for the EHR system. Most hospitals define a “short” outage as an hour or less, followed by an escalating set of protocols that work their way up to a “long-term” outage of 12 to 18 hours at the most. That approach measures potential outages in hours, but our experience, and the recent experiences of other hospitals, shows that contingency plans should redefine long-term outages to be measured in days or even weeks, in order to adequately prepare for what might happen if your EHR software goes down.

In addition to the extended length of our outage, there were three additional aspects of the situation that are important to discuss:

Page
of 5Next
Topics

Comments

extended EHR outage

In the world of Financial IT, it has long been required that there be adequate backup equipment and backup'd data. A typical situation; there is a site in another power grid and with somewhat separate telecom capability where the 'production systems can be brought back up. Data ought to be backed up, if not continuously, then at least frequently to the 2nd site and perhaps to a purely data backup facility as well. At any given moment, if the primary computing facility goes down, the backup can take over quickly with up to or almost up to date data. The site where the people are, a bank - or a hospital, has to have backup power and telecom capabilities, and perhaps even backup workstations.

HealthIT should be no different - there's just as much at stake if not more. The days of having one little computer room in the hospital basement and someone trying to remember to take home backup tapes once in a while need to be far in our past. If the EHR primary system goes down, the secondary needs to kick in. Period. This, by the way, is one of a number of arguments for 'cloud' based systems.

I hope this never happens to you

This took a pretty big toll on our IT Department and our Board, staff, medical staff and senior leadership. I hope there are some lessons learned in here for you and your organization.

Sad

After reading this article, you might get the feeling that Linda actually cares about the people that work for her. The reality is, that in her mind, every person beneath her is disposable and easily replaceable. There is a line of 'bodies' a mile long, that have been chewed up and spit out by this poor excuse for a leader. Do not be misled by a well written article that was written for no other reason than furthering her career.

It is unfortunate that you feel this way

Dear Anonymous Sad RM460

It is very disappointing that you took this important “Lessons Learned” article as an opportunity to voice your personal opinion and dissatisfaction in such a public and unprofessional manner.

I am sorry to hear that you are disgruntled but I would be more than happy to meet with you in person to hear your concerns directly. Also, as you know, employees at BCH are always welcome to seek the assistance of our Human Resources Department.

Feel free to e-mail me directly at lminghella@bch.org if you would be interested in a follow up discussion about your dissatisfaction.