Christian points to that as an example of how a relatively minor repair in the data center was quickly able to spiral out of control. But he also cautions that potential catastrophes can just as easily be mundane. “When you start talking about disaster recovery, it's much more than what happens in the data center; it's a facility thing,” he says. Good Samaritan Hospital, which is located on a 100-year flood plain, has embarked on a new building program, and will be vacating a tower. The IT department will relocate from its current location, known as The Pit because it is the lowest point in the facility-to the third floor of the old tower, well above the flood plain.
Diana Boyer, R.N., vice president and CIO of Columbus Regional Hospital, located in Columbus, Ind., about 45 miles south of Indianapolis, would agree. In June of 2008, the Columbus area experienced 11 inches of rain in a very short period of time, causing a small nearby stream to swell so rapidly that it resulted in a flash flood.
In just 45 minutes, the stream overflowed its banks and spilled into the hospital parking lot, making its way toward the hospital basement that housed the data center, electrical switches, laboratory, food services, pharmacy, and other core services. The water reached the below-grade loading dock, which acted as a funnel that flooded the 160,000-square-foot basement in less than an hour. The flood submerged the electrical switches, causing the hospital to lose power, and then began to bubble up to the 140,000 square-foot main floor, putting the radiology department and ED out of commission. The hospital staff successfully evacuated the hospital's 157 patients without injuries, Boyer says.
Boyer was home and couldn't get to the hospital. She called her staff at 4 p.m. and was told everything was fine; but by 5:30, the hospital had shut down. Estimated damages were close to $200 million and closure was estimated for 12 to 18 months.
In the January prior to the flood, Columbus Regional had acquired a data center three miles away, and was in the process of installing redundant systems off site, which until then existed in the original data center that was now under water. The original data center also housed two storage area networks, one of which was to be moved to the new data center; and plans were to install a second uninterruptable power supply and backup generator. None of the installations were completed before the flood, Boyer says.
The original data center was destroyed in short order, with its equipment immersed in creek water, mud, corn stalks from local fields, reagents from the submerged lab, and other contaminants that made the computer drives useless. Fortunately, the hospital kept tape backups of data at another location in Indianapolis, so there was minimal data loss, Boyer says. The flood destroyed the hospital's phone system, forcing the staff to rely on cell phones.
Remarkably, Columbus Regional was operational less than five months after the flood. The Carolinas MED-1 mobile emergency department unit arrived June 23. MED-1 provided emergency care for the area until the hospital opened the ED in the main building on Aug. 1. On Oct. 27, it re-opened patient care services, including surgery. Actual damages totaled $171 million.
Today the hospital's primary data center is housed in the new facility three miles off site, and the secondary data center is located on the first floor of the main hospital. The electrical switches have been moved out of the basement, as have the pharmacy and laboratory, which are now on the hospital's main floor.
Boyer places a lot of value on the hospital's expert IT staff, which was hired in 1997 to develop the IT department as part of the hospital's strategic IS plan. “We did not contract out implementations or technical support. We kept the knowledge here at Columbus Regional,” she says. “We had a very structured methodology on how we implemented systems; designing, building, testing and training; and how we keep it. Good relationships with vendors also were important,” she says.
TRADITIONAL DATA CENTERS ARE GOING TO START TO HAVE A MUCH SMALLER FOOTPRINT THAN THEY DO TODAY, AND WE ARE GOING TO START TO SEE THEM GO AWAY.-RUSSell p. BRANZELL
Just prior to the flood, Columbus Regional participated in a multi-county disaster recovery drill based on a potential terrorism incident. The hospital is now expanding on the command center model that was developed then. One of the biggest lessons of the flood had to do with the role reversal: hospitals traditionally receive disaster victims, but are not usually victims themselves. “That was a huge turnaround for us,” she says.
REACHING UP TO THE CLOUD
Interestingly, even though Columbus Regional has successfully completed and relocated its new data centers, which it operates, it plans to move to remote hosting of its electronic health record (EHR) system when the hospital moves to Kansas City-based Cerner Corp. from its current vendor next summer. (The hospital will continue to host its own PACS and financial system.) Boyer says she is comfortable with the decision, noting that the hospital has successfully hosted its lab system remotely with Cerner for six years. Financially, remote hosting makes sense, saving the hospital the costs of buying and maintaining its own hardware and maintaining an Oracle database, she says.
