Every day, Kindred’s information system (IS) staff is proud to say we work as if people’s lives depended on it. That’s because technology plays a critical role in a healthcare environment, helping facilities deliver high-quality patient care and outstanding customer service. The top priority is that all data, from financial processing to clinical reports, is available online from anywhere in the enterprise.
Kindred is a healthcare services company that, through its subsidiaries, operates hospitals, nursing centers and a rehabilitation services business across the United States. As new companies are acquired, additional demands are placed on the IT infrastructure, such as supporting more users, company codes and ad-hoc reporting requests.
The IS department supports over 1,000 software applications, including an integrated financial, human resources and payroll system and business intelligence data warehouse. There are more than 1,300 servers connected by a robust frame relay network to 16,000 workstations supporting about 350 IS professionals.
Overall, Kindred’s employee base has risen to more than 56,000 employees spread across 138 active company codes. IT infrastructure has been responding to this growth, as evidenced by the database expanding 10-fold over eight years, from 200 gigabytes to 2 terabytes.
The company’s growth was making it difficult to complete month-end activities on time, leading it to upgrade its five-year-old server landscape with new high-performance servers. Subsequently the IS department ran a series of before-and-after benchmark tests and calculated the difference in system performance. We measured the time to run financial and HR reports, SAP transactions, overnight batch jobs, as well as online user response time. The upgrade provided a significant increase in speed, ranging from 24 percent to 800 percent, depending on the application and the number of users.
Unable to scale older machines along with the pace of the company’s expansion, the IS staff observed the strain on the system. It became more and more difficult to close the books within a 10 day window after month-end, an IT department service-level commitment. The database server experienced 100 percent utilization for two hours every morning during month-end close, as well as usage spikes that consumed all of the system capacity throughout the day. End-user online response time averaged above two seconds, with some processes experiencing severe delays. Night time batch and payroll runs were running into the morning hours and affecting online users.
Beyond addressing these performance issues, Kindred’s key objective was to find a solution that could accommodate new business and acquisitions, improve SAP application performance and increase uptime (high availability). Another requirement was a cost-effective solution, providing return on investment (ROI) in five years or less.
We sought the help of HP to architect a new scalable IT infrastructure that would support our SAP R/3 software and anticipated business growth. This meant increasing capacity, improving response time and increasing system availability. These objectives were met through two significant upgrades.
First, 15 legacy servers were replaced with nine HP Integrity servers (rx7640 and rx4640) with Intel Itanium processors, migrating from 32-bit systems to 64-bit to increase performance. These systems run 64-bit application software, including mySAP ERP 2005, Microsoft Windows Server 2003 operating system and Microsoft SQL Server 2000, which allows users to query, analyze and manipulate data over the Web.
Second, the servers were organized in clusters — pools of independent servers that work together as a single system. Server clusters can increase capacity and improve response time by allowing the software to spread the workload among multiple servers. This helps avoid having one server 100 percent utilized while other servers are underutilized. There are also highly available clusters consisting of two or more nodes that are exact mirror images of each other. If the primary system goes down due to hardware malfunction, fail-over software immediately makes its twin system the primary node.
With a project of this size, there are usually a number of decisions and implementation processes that have a significant impact on the outcome. One such decision was to contract HP Services engineers to rack and configure the servers before they were sent to us. They installed the OS Image and software we sent them. In addition, they configured IP addresses, system names, worldwide names for the SANs and other system parameters. Our alternative was to hire three or four external heads and probably take a several week schedule hit. Afterward, HP documented the configuration process and burned CDs of the server images.
We also decided not to wait for a server refresh that incorporated the latest dual-core Intel Itanium and the HP sx2000 chipset. These newer servers would have given us almost twice the performance of the single-core processor versions we implemented. Given our aggressive project schedule to remedy our seriously overtaxed infrastructure, such delays could not be accommodated.
By adding a keyboard, video and mouse (KVM) switch that allows multiple servers to share a console, system maintenance has become easier and more convenient. Administrators can remotely VPN into the network, connect to the switch and access the various servers.
Some project aspects were unpleasant. Getting every printer ready for the migration was one of the biggest headaches and time syncs. We had to clean up and test all the printer configurations in advance. And although annual server maintenance costs were known upfront, they are surprisingly high.
The most valuable implementation lesson involves early participation by end-users. As soon as the system was ready, many users were brought in to perform regression testing over a four-week period. This provided before and after data on every business process — by running through all the transactions — and made sure the end results and speed were as expected. By getting users to check all the functionality upfront, we also got their early buy-in into the new system.
Documenting everything is also very important during implementation. We captured every installation and configuration detail in a 120-page document.
The I/S department ran a series of benchmark tests that measured the performance improvement resulting from the server and software upgrades.
Financial and HR Reports: Kindred’s financial group runs 60 reports — which consumed 100 percent of the older database machines — during the month-end close (eight servers in all). This load made the system unresponsive to online users. The new database configuration consisted of two HP servers that ran all 60 reports in parallel, finishing them at the same time, with a peak CPU utilization of 60 percent. General ledger, 10K and capital reports finished 2.5 times faster, on average. Three HR reports, which take between seven and 23 hours to run, were more than three times faster, on average.
SAP Transactions: We measured the response time for six heavily used SAP transactions: three standard and three custom. The new system executed these transactions an average of 1.8 times faster.
Overnight Batch Jobs: Batch processing time was extending into the morning hours and impacting day workers. We tested a large composite production control batch, a superset of processes that normally run between 5 p.m. and 7 a.m. The legacy system required about 25 hours while the new system took only 19 hours, equaling a 24 percent time reduction.
Online User Response Time: Response time was measured while the system was loaded with 1,600 users. Wait times for application dialog processes dropped from over two seconds to less than half a second, enabling online users to flip through screens faster. Application software reading the database responded six times faster.
The new servers were deployed quickly and with little impact to operations. As a result of the server consolidation efforts, server administration requires less effort. The upgrade doubled capacity and scalability with six fewer servers, which eliminated the 10-day month-end close issues from the previous platform. Kindred already received its ROI from this project — the company will save $1 million over five years by avoiding the expense of forklifting the entire server infrastructure.
Daniel Poff is the director of database administration and SAP Basis for Kindred Healthcare in Louisville, Ky.