When Disaster Strikes: How Technology Drives Better Preparation

The hospital was already live with its EHR (supplied by Epic Systems Corp., Verona, Wis.), which was fully functional when the tornado struck. Had it struck prior to the go-live, it would have been much worse from a data standpoint, McCreary says: “We would have lost all of the systems; and even though there were backups, once something like that happens you are restoring new equipment, and there are always complications.”

McCreary notes that the data center in Washington, which was up and running at the time, works with a failover site in St. Louis. He adds that there is also a local component for each of the hospitals, where local servers would be used for storage and faster access. Those were destroyed at Joplin.

In the short term, the availability of patient data is crucial, he says. During recovery, “You want as much detail around that type of information, because you are dealing with casualties first off, and the more of that information you have available, the better care everybody gets in the beginning,” he says. After that, the hospital moved as quickly as possible to get back to a normal mode.

“Our goal is one patient, one record, and that record should include every encounter he has ever had with a Mercy hospital or clinic,” McCreary says. Since the Joplin tornado, Mercy has hardened its data center in Washington, adding that it centralizes data as much as possible.

In addition to running its own data centers, Mercy is also a data center vendor, selling disaster recovery advisory services to other health organizations, McCreary says. He also thinks that large healthcare organizations should look for ways to work together to back each other’s data.

Florida Hospital Goes Virtual

Florida Hospital, part of the Adventist Health System, is a 2,247-bed acute care organization with seven campuses in the Orlando metropolitan area. In 2004, the hospital experienced a “hurricane trifecta”—three hurricanes in one year—according to Robert Goodman, the hospital’s disaster recovery coordinator. At the time, the hospital used a “tapes and trucks” process, in which data backup tapes were transported physically to a secondary data center.

Although the hospital did not declare a disaster that season, had the data center been destroyed, the backed up data from the tapes would have been several days old, Goodman says. This prompted the hospital to explore alternatives for backing up data, which are centered on remote replication and virtualization.

In 2006, the hospital abandoned the use of backup tapes in favor of continuously replicating data to a secondary data center (operated by SunGard Availability Services, Wayne, Pa., which provides all data recovery services to the hospital) located nearly 1,000 miles from the hospital’s primary data center—and a safe distance from regional disasters. “Basically we mirror asynchronously to our hot site, and that keeps our data current,” Goodman says.

Robert Goodman

In addition, Goodman reports that Florida Hospital has begun to virtualize its environments in both its home data center and its secondary site. Virtualization of its servers allows applications to be deployed very quickly, because they are not tied to a specific piece of hardware, he says, adding that virtualization also provides scalability, an important factor because the hospital has more than 100 disaster recovery applications. “We’re getting scalability because we don’t have as many physical servers, and replication is quick,” he says.

The hospital conducts a business impact analysis to classify which systems get backed up. “We look at what the impact would be to the enterprise if those applications were down for a certain period of time,” he says. The applications are tiered accordingly.

Goodman says data is written to disk, which is then replicated to the secondary site. With remote replication using virtualized servers, data replication time has gone down by a factor of 10. “That’s a good thing, because underlying technologies are becoming more complex,” he says.

In addition, Goodman says the hospital has begun to use remote access for his IT staff, which he calls “virtualizing the workforce.” Using an encrypted virtual private network (VPN) connection over the Internet, IT staff can access the recovery site and the home site remotely. At one point, Florida Hospital required 19 individuals to be on site during a recovery exercise. During its last exercise it sent four technicians; and in the next two years he hopes to be able to work 100 percent remotely.