CIOs are hard at work coming up with the most effective and affordable strategies for protecting electronic data as their hospitals move forward on electronic medical records. While the rise of cloud computing and declining network costs are offering new opportunities in dealing with potential disasters, many find there is no substitute for good planning and constant testing.
Ask any hospital CIO what keeps him or her up at night, and chances are that disaster preparedness ranks high on their lists. In fact, as this issue was about to go to press, Hurricane Irene roared up the Eastern Seaboard, causing massive flooding in coastal cities and towns from the Carolinas to Maine. As if to underline the seriousness of the threat, New York City officials took the unprecedented step of shutting down that city's mass transit system and ordering the evacuation of four major hospitals that were located in flood areas.
To be sure, hospitals across the country are facing serious financial pressures as they push forward on meeting meaningful use benchmarks under the American Recovery and Reinvestment Act/Health Information Technology for Economic and Clinical Health (ARRA-HITECH) Act. Yet despite their tight budgets, CIOs interviewed for this article also indicate that they have gone to great lengths to make sure that they have plans in place that will help hospitals continue to function if disaster strikes.
Not least, a catastrophic failure could threaten the progress that hospitals have already made toward clinical decision support and instant access to clinical information that electronic health records have made possible, notes Charles E. Christian, CIO of Good Samaritan Hospital, Vincennes, Ind. As the CIO of a county hospital in a rural area, he says it's essential to have a good handle on what the hospital needs and what it can afford. “It's like an insurance premium to make sure you can move your backup tapes and get those taken care of,” he says.
He compares disaster recovery planning to Y2K planning at the end of the last century. “We did all of the remediation and Y2K was a non-event. It's the same type of planning we have with disaster recovery and business continuity planning. We want to make sure that nothing happens,” he says.
A look at the strategies that hospitals are pursuing reveals that they are often an interesting mix of current IT trends, such as the cloud, and traditional brick-and-mortar issues.
SPREADING THE RISK
Chuck Podesta, senior vice president and CIO of Fletcher Allen Health Care, Burlington, Vt., views disaster recovery as a life safety issue, similar to certain types of medical equipment that need to be maintained. Virtualization, network redundancy, multiple data centers, and the cloud all figure in preparation. “You are spreading your data over multiple geographic areas, which is part of the high availability strategy, and is extremely important these days to look at,” he says.
The key challenge for each CIO is the need to assess the risk that their hospital faces and coming up with a plan that is appropriate to that level of risk, according to Russell P. Branzell, vice president and CIO of Poudre Valley Health System in Fort Collins, Colo. “Are you really in an area that is highly prone to disaster, and are you able to take the action that is appropriate to the level of risk?”
His hospital's disaster recovery process “is not something that collects dust on a shelf for me as a CIO,” Branzell says. “I get a new binder every year from my security and technical team; they've gone through the process for disaster recovery, and then I put recovery audit in every two years. It gives me the confidence that I have a fighting chance if something bad was to happen.”
Although Poudre Valley does not have truly mirrored sites-an arrangement in which every piece of data is continuously and fully replicated-it does have basic functionality for business continuity, with good backup procedures and de-duplication processes in place. Data is stored on-site at a secure location, Branzell says.
DISASTERS LARGE AND SMALL
Despite the best of precautions, no hospital can completely eliminate the possibility of a failure, usually unexpected, and sometimes resulting from an unlikely daisy-chain of events. At Good Samaritan recently, a transfer switch failed in the power room. During the repair operation, the uninterruptable power supply (UPS) that was providing backup power also failed. This disrupted power to the DNS servers that handled directory information for all of the applications, which then could not connect with the databases, which could not get to the storage area network. “We spent hours recovering that, and we were up in three hours,” Christian says, adding that the IT team also had to check to make sure that the databases were not corrupted.
WE DID NOT CONTRACT OUT IMPLEMENTATIONS OR TECHNICAL SUPPORT. WE KEPT THE KNOWLEDGE AT COLUMBUS REGIONAL. -DIANA BOYER, R.N.