EXECUTIVE SUMMARY:

CIOs are hard at work coming up with the most effective and affordable strategies for protecting electronic data as their hospitals move forward on electronic medical records. While the rise of cloud computing and declining network costs are offering new opportunities in dealing with potential disasters, many find there is no substitute for good planning and constant testing.

Ask any hospital CIO what keeps him or her up at night, and chances are that disaster preparedness ranks high on their lists. In fact, as this issue was about to go to press, Hurricane Irene roared up the Eastern Seaboard, causing massive flooding in coastal cities and towns from the Carolinas to Maine. As if to underline the seriousness of the threat, New York City officials took the unprecedented step of shutting down that city's mass transit system and ordering the evacuation of four major hospitals that were located in flood areas.

To be sure, hospitals across the country are facing serious financial pressures as they push forward on meeting meaningful use benchmarks under the American Recovery and Reinvestment Act/Health Information Technology for Economic and Clinical Health (ARRA-HITECH) Act. Yet despite their tight budgets, CIOs interviewed for this article also indicate that they have gone to great lengths to make sure that they have plans in place that will help hospitals continue to function if disaster strikes.

Not least, a catastrophic failure could threaten the progress that hospitals have already made toward clinical decision support and instant access to clinical information that electronic health records have made possible, notes Charles E. Christian, CIO of Good Samaritan Hospital, Vincennes, Ind. As the CIO of a county hospital in a rural area, he says it's essential to have a good handle on what the hospital needs and what it can afford. “It's like an insurance premium to make sure you can move your backup tapes and get those taken care of,” he says.

He compares disaster recovery planning to Y2K planning at the end of the last century. “We did all of the remediation and Y2K was a non-event. It's the same type of planning we have with disaster recovery and business continuity planning. We want to make sure that nothing happens,” he says.

A look at the strategies that hospitals are pursuing reveals that they are often an interesting mix of current IT trends, such as the cloud, and traditional brick-and-mortar issues.

SPREADING THE RISK

Chuck Podesta, senior vice president and CIO of Fletcher Allen Health Care, Burlington, Vt., views disaster recovery as a life safety issue, similar to certain types of medical equipment that need to be maintained. Virtualization, network redundancy, multiple data centers, and the cloud all figure in preparation. “You are spreading your data over multiple geographic areas, which is part of the high availability strategy, and is extremely important these days to look at,” he says.

The key challenge for each CIO is the need to assess the risk that their hospital faces and coming up with a plan that is appropriate to that level of risk, according to Russell P. Branzell, vice president and CIO of Poudre Valley Health System in Fort Collins, Colo. “Are you really in an area that is highly prone to disaster, and are you able to take the action that is appropriate to the level of risk?”

His hospital's disaster recovery process “is not something that collects dust on a shelf for me as a CIO,” Branzell says. “I get a new binder every year from my security and technical team; they've gone through the process for disaster recovery, and then I put recovery audit in every two years. It gives me the confidence that I have a fighting chance if something bad was to happen.”

Although Poudre Valley does not have truly mirrored sites-an arrangement in which every piece of data is continuously and fully replicated-it does have basic functionality for business continuity, with good backup procedures and de-duplication processes in place. Data is stored on-site at a secure location, Branzell says.

DISASTERS LARGE AND SMALL

Despite the best of precautions, no hospital can completely eliminate the possibility of a failure, usually unexpected, and sometimes resulting from an unlikely daisy-chain of events. At Good Samaritan recently, a transfer switch failed in the power room. During the repair operation, the uninterruptable power supply (UPS) that was providing backup power also failed. This disrupted power to the DNS servers that handled directory information for all of the applications, which then could not connect with the databases, which could not get to the storage area network. “We spent hours recovering that, and we were up in three hours,” Christian says, adding that the IT team also had to check to make sure that the databases were not corrupted.

WE DID NOT CONTRACT OUT IMPLEMENTATIONS OR TECHNICAL SUPPORT. WE KEPT THE KNOWLEDGE AT COLUMBUS REGIONAL. -DIANA BOYER, R.N.

Christian points to that as an example of how a relatively minor repair in the data center was quickly able to spiral out of control. But he also cautions that potential catastrophes can just as easily be mundane. “When you start talking about disaster recovery, it's much more than what happens in the data center; it's a facility thing,” he says. Good Samaritan Hospital, which is located on a 100-year flood plain, has embarked on a new building program, and will be vacating a tower. The IT department will relocate from its current location, known as The Pit because it is the lowest point in the facility-to the third floor of the old tower, well above the flood plain.

Diana Boyer, R.N., vice president and CIO of Columbus Regional Hospital, located in Columbus, Ind., about 45 miles south of Indianapolis, would agree. In June of 2008, the Columbus area experienced 11 inches of rain in a very short period of time, causing a small nearby stream to swell so rapidly that it resulted in a flash flood.

In just 45 minutes, the stream overflowed its banks and spilled into the hospital parking lot, making its way toward the hospital basement that housed the data center, electrical switches, laboratory, food services, pharmacy, and other core services. The water reached the below-grade loading dock, which acted as a funnel that flooded the 160,000-square-foot basement in less than an hour. The flood submerged the electrical switches, causing the hospital to lose power, and then began to bubble up to the 140,000 square-foot main floor, putting the radiology department and ED out of commission. The hospital staff successfully evacuated the hospital's 157 patients without injuries, Boyer says.

Boyer was home and couldn't get to the hospital. She called her staff at 4 p.m. and was told everything was fine; but by 5:30, the hospital had shut down. Estimated damages were close to $200 million and closure was estimated for 12 to 18 months.

In the January prior to the flood, Columbus Regional had acquired a data center three miles away, and was in the process of installing redundant systems off site, which until then existed in the original data center that was now under water. The original data center also housed two storage area networks, one of which was to be moved to the new data center; and plans were to install a second uninterruptable power supply and backup generator. None of the installations were completed before the flood, Boyer says.

The original data center was destroyed in short order, with its equipment immersed in creek water, mud, corn stalks from local fields, reagents from the submerged lab, and other contaminants that made the computer drives useless. Fortunately, the hospital kept tape backups of data at another location in Indianapolis, so there was minimal data loss, Boyer says. The flood destroyed the hospital's phone system, forcing the staff to rely on cell phones.

Remarkably, Columbus Regional was operational less than five months after the flood. The Carolinas MED-1 mobile emergency department unit arrived June 23. MED-1 provided emergency care for the area until the hospital opened the ED in the main building on Aug. 1. On Oct. 27, it re-opened patient care services, including surgery. Actual damages totaled $171 million.

Today the hospital's primary data center is housed in the new facility three miles off site, and the secondary data center is located on the first floor of the main hospital. The electrical switches have been moved out of the basement, as have the pharmacy and laboratory, which are now on the hospital's main floor.

Boyer places a lot of value on the hospital's expert IT staff, which was hired in 1997 to develop the IT department as part of the hospital's strategic IS plan. “We did not contract out implementations or technical support. We kept the knowledge here at Columbus Regional,” she says. “We had a very structured methodology on how we implemented systems; designing, building, testing and training; and how we keep it. Good relationships with vendors also were important,” she says.

TRADITIONAL DATA CENTERS ARE GOING TO START TO HAVE A MUCH SMALLER FOOTPRINT THAN THEY DO TODAY, AND WE ARE GOING TO START TO SEE THEM GO AWAY.-RUSSell p. BRANZELL

Just prior to the flood, Columbus Regional participated in a multi-county disaster recovery drill based on a potential terrorism incident. The hospital is now expanding on the command center model that was developed then. One of the biggest lessons of the flood had to do with the role reversal: hospitals traditionally receive disaster victims, but are not usually victims themselves. “That was a huge turnaround for us,” she says.

REACHING UP TO THE CLOUD

Interestingly, even though Columbus Regional has successfully completed and relocated its new data centers, which it operates, it plans to move to remote hosting of its electronic health record (EHR) system when the hospital moves to Kansas City-based Cerner Corp. from its current vendor next summer. (The hospital will continue to host its own PACS and financial system.) Boyer says she is comfortable with the decision, noting that the hospital has successfully hosted its lab system remotely with Cerner for six years. Financially, remote hosting makes sense, saving the hospital the costs of buying and maintaining its own hardware and maintaining an Oracle database, she says.

The decision is part of a trend in which cloud providers, application service providers, and other third-party relationships are becoming an increasingly important partner with hospital IT departments when it comes to storing and processing data and housing hardware.

Case in point: Good Samaritan is a member of the Indiana Network for Patient Care (INPC) in Indianapolis, part of a service for the Indiana Health Information Exchange that acts as a repository for clinical information. When an HL7 registration for a new patient hits the INPC server, it returns clinical histories for each individual patient, providing the physician with the patient's history. Christian says Good Samaritan is now planning on using the service as a disaster recovery tool, which he describes as a secondary on-line ready access to the EMR. “We are already paying for the service, so we can add value to that expense by providing access to a clinical record in case we have a failure,” he says.

Poudre Valley's Branzell believes the cloud is going to be a game-changer for the entire applications market. “Traditional data centers are going to start to have a much smaller footprint than they do today, and we are going to start to see them go away,” he says.

That's not to say that the trend does not bring its own concerns. Every healthcare data center in the country has some single point of failure, Branzell says. In his own experience, he realized that Poudre Valley had dual grid power, both going through the same switch. “No matter what you do, there is some place where you have got something goofy, where you have got a single point of failure,” Branzell says.

Single point of failure is becoming increasingly important as more solutions become cloud-based or application specific provider-(ASP) based, Branzell says. “If all of my transcription services are done in Boston via a cloud solution or a Web solution, and I can't get to that, how do I do my transcription?” he asks. That concern has led him to seek lower bandwidth alternative paths to its Internet service provider that can be used as a failover. Poudre Valley's imaging is an ASP-based solution through Phillips; however, it also stores 90 days worth of images locally, as a backup.

Bruce Smith, senior vice president and CIO of Advocate Health Care, Oak Brook, Ill., notes that the IT environment has become extremely complex over the last 30 years. In October 2009 the organization was evaluating a hardware and software upgrade that was going to cost in the $20 million range, and decided to contract with Cerner's cloud services, including its main EMR, physician order entry, clinical information retrieval, laboratory results and radiology results, and disaster recovery. “All clinical information for inpatient, outpatient, and the ED is maintained through this system,” he says. Advocate still maintains some systems separately, including registration and billing, as well as the EMR for its physician group and independent physician offices.

Smith says the decision to use the cloud was based on a combination of business issues, and the timing of the upgrade was one of the deciding factors in deciding to use Cerner's cloud service. “We asked ourselves if we really wanted to make this hardware investment that we would have to replenish in 10 years, or just bundle the stuff into an agreement,” he says. The hardware covered in the agreement includes a Citrix server farm, which Smith says is difficult to maintain, and a secure state-of-the-art data center that is protected against class 6 tornados. Cerner's two data centers are located 20 miles apart in the Kansas City area, one for primary use and the other providing backup.

When Smith felt the hospital had a good basic outsourcing agreement, he got into specifics of disaster recovery. The agreement includes redundancy, but not full redundancy, in which the standby environment would provide immediate backup. “If we have a problem with our main database and processing, we would do a switchover, which is guaranteed to be up in four hours,” he says. Full redundancy with immediate switchover would be significantly more expensive, although going that route is a possibility in the future, he says.

“The most critical data is the clinical data physicians use to treat their patients,” Smith says. “We put a lot of effort into making sure that the front-end connection is reliable and works well, and that has been the case.” In addition, Smith says, Advocate has a number of communication pipelines connecting the data centers to the Chicago area, where the hospital is located, so if there is a problem with one of the communication links, it will fail to another. “We have complete redundancy in data communication from here to Kansas City,” he says.

Fletcher Allen, whose EHR is from Epic Systems Corp., Verona, Wis., operates three data centers. “We built in redundancy by doing a lot of shadowing of data across the data centers,” Podesta says. He says running one's own data center is “very expensive. Keeping the network up to date and managing that takes up a good piece of the annual budget.”

He says transferring that responsibility to third-party vendors is a possibility in the future, especially considering the escalating costs of cooling, electricity and maintenance of the space itself. It is a possibility that he looks at each year at budget time. “In the past, technology has not been up to snuff, and the bandwidth you would need to do that would be very costly,” Podesta says. “But all of that is coming down now; the price of hardware, storage, and the networks themselves, and you can get a lot of bandwidth at a lower cost than in the past.” In his view, cloud-based EHRs will begin to become more common in three to five years as the technology becomes more robust and vendors become more focused on it.

Not everyone is convinced that the cloud has enough of a track record for hospital CIOs to give up direct control of their data centers. Steve Duch, director of IS business development at Memorial Sloan-Kettering Cancer Center in Manhattan, says that “the cloud has not yet been tested in the event of an actual major disaster.”

While the question of how big a role the cloud will play in disaster recovery plans remains open to debate, it's clear that progress toward electronic health records is both offering hospitals new tools and exposing them to new and different vulnerabilities in addition to longstanding natural threats.

Healthcare Informatics 2011 October;28(10):16-21