|
Reprints
DISASTERS LARGE AND SMALL
Despite the best of precautions, no hospital can completely eliminate the possibility of a failure, usually unexpected, and sometimes resulting from an unlikely daisy-chain of events. At Good Samaritan recently, a transfer switch failed in the power room. During the repair operation, the uninterruptable power supply (UPS) that was providing backup power also failed. This disrupted power to the DNS servers that handled directory information for all of the applications, which then could not connect with the databases, which could not get to the storage area network. “We spent hours recovering that, and we were up in three hours,” Christian says, adding that the IT team also had to check to make sure that the databases were not corrupted.
WE DID NOT CONTRACT OUT IMPLEMENTATIONS OR TECHNICAL SUPPORT. WE KEPT THE KNOWLEDGE AT COLUMBUS REGIONAL. -DIANA BOYER, R.N.
Christian points to that as an example of how a relatively minor repair in the data center was quickly able to spiral out of control. But he also cautions that potential catastrophes can just as easily be mundane. “When you start talking about disaster recovery, it's much more than what happens in the data center; it's a facility thing,” he says. Good Samaritan Hospital, which is located on a 100-year flood plain, has embarked on a new building program, and will be vacating a tower. The IT department will relocate from its current location, known as The Pit because it is the lowest point in the facility-to the third floor of the old tower, well above the flood plain.
Diana Boyer, R.N., vice president and CIO of Columbus Regional Hospital, located in Columbus, Ind., about 45 miles south of Indianapolis, would agree. In June of 2008, the Columbus area experienced 11 inches of rain in a very short period of time, causing a small nearby stream to swell so rapidly that it resulted in a flash flood.
In just 45 minutes, the stream overflowed its banks and spilled into the hospital parking lot, making its way toward the hospital basement that housed the data center, electrical switches, laboratory, food services, pharmacy, and other core services. The water reached the below-grade loading dock, which acted as a funnel that flooded the 160,000-square-foot basement in less than an hour. The flood submerged the electrical switches, causing the hospital to lose power, and then began to bubble up to the 140,000 square-foot main floor, putting the radiology department and ED out of commission. The hospital staff successfully evacuated the hospital's 157 patients without injuries, Boyer says.
Boyer was home and couldn't get to the hospital. She called her staff at 4 p.m. and was told everything was fine; but by 5:30, the hospital had shut down. Estimated damages were close to $200 million and closure was estimated for 12 to 18 months.
In the January prior to the flood, Columbus Regional had acquired a data center three miles away, and was in the process of installing redundant systems off site, which until then existed in the original data center that was now under water. The original data center also housed two storage area networks, one of which was to be moved to the new data center; and plans were to install a second uninterruptable power supply and backup generator. None of the installations were completed before the flood, Boyer says.
The original data center was destroyed in short order, with its equipment immersed in creek water, mud, corn stalks from local fields, reagents from the submerged lab, and other contaminants that made the computer drives useless. Fortunately, the hospital kept tape backups of data at another location in Indianapolis, so there was minimal data loss, Boyer says. The flood destroyed the hospital's phone system, forcing the staff to rely on cell phones.
Remarkably, Columbus Regional was operational less than five months after the flood. The Carolinas MED-1 mobile emergency department unit arrived June 23. MED-1 provided emergency care for the area until the hospital opened the ED in the main building on Aug. 1. On Oct. 27, it re-opened patient care services, including surgery. Actual damages totaled $171 million.
Today the hospital's primary data center is housed in the new facility three miles off site, and the secondary data center is located on the first floor of the main hospital. The electrical switches have been moved out of the basement, as have the pharmacy and laboratory, which are now on the hospital's main floor.
Boyer places a lot of value on the hospital's expert IT staff, which was hired in 1997 to develop the IT department as part of the hospital's strategic IS plan. “We did not contract out implementations or technical support. We kept the knowledge here at Columbus Regional,” she says. “We had a very structured methodology on how we implemented systems; designing, building, testing and training; and how we keep it. Good relationships with vendors also were important,” she says.
