It has also created an environment that, while dynamic, does require upkeep when new IT applications are added to the hospital. “When new systems come on board, we add storage and we add servers in the virtual environment,” Evans says. “This way, everything has a backup strategy if it fails.”

Layers of protection

For Carilion Clinic, an eight-hospital, 1,125-bed organization, planning an aggressive Epic EMR rollout meant reassessing data storage capabilities. “You can't go to an electronic record unless you have a fail-safe IT backbone on which to run it,” says Daniel Barchi, senior vice president and CIO of the Roanoke, Va.-based system. “We thought that was important - so important, in fact, that in addition to our primary data center, we built a secondary data center as we were rolling out the EMR.”

According to Barchi, Carilion's data centers have dual network feeds, power systems and back-up systems, ensuring not only that each individual system can continue operations in the event of an emergency, but also that if one is completely down, operations can switch to the other. “This way, we can run our IT operations in parallel and we can failover from one to another, so we're never single-threaded through a single data center,” he says.

And while Carilion's data centers and EMR haven't experienced any downtime since the first sites went live in February of 2008, Barchi and his staff still feel it is critical to have a two-pronged approach to ensure that patient information is accessible even if servers are unavailable. If the hospital still has Internet access, clinicians can view data using read-only servers, and if all connectivity is lost, they can use business continuity computers on which all information is stored locally, providing a “continuously updated snapshot of what's happening with every patient,” says Barchi. “We have about four layers of protection. Your system can stay up, but if your network goes down, that's still a problem. That's why we've added those additional layers.”

Test and test again

Simply having a data protection strategy in place, however, isn't enough, according to Weiner, who says periodic testing should play a key role in the CIO's server reliability plan. “Testing is critical to make sure that whatever you've deployed actually works,” he says, recommending that solutions are tested at least annually, if not on a semiannual or quarterly basis.

Thibodaux Regional tests its disaster recovery system once a year; in-house servers, however, are pulled down every 90 days for maintenance, which Evans says helps sustain a dynamic environment.

Carilion's staff schedules downtimes during off-hours to install upgrades. Barchi says nurses and clinicians can either use the read-only system or the local backup system to continue documenting patient care.

But while testing is a critical piece of the strategy, Podesta urges CIOs to learn from his organization's incident and go one step further by trying to anticipate things.

“The problem is, when you test, it's in a controlled environment. If you really want to test your system, you need to think of various scenarios that might happen and try to mimic them in some way,” says Podesta.

And while he couldn't have foreseen that Fletcher Allen would lose its EMR for several hours just months after going live, Podesta says he was aware that downtimes are possible. “We incorporated the business continuity aspect and downtime procedures into the training process, so I think that helped out. But going forward, we'll probably do more testing than we originally planned.”

Though some leaders might balk at the costs involved in achieving such a high level of availability and redundancy, the investment is well worth reducing the risk associated with a potential downtime, says Weiner. “For every day that systems are down, it can decrease a hospital's cash flow and increase expenses,” he says.

For Evans, the costs related to duality and recovery all are part of the package. “It's part of the decision when you select a clinical system,” he says. “Because once you get into that environment, you can't go down - the hospital is out of business.”