Every day, Kindred’s information system (IS) staff is proud to say we work as if people’s lives depended on it. That’s because technology plays a critical role in a healthcare environment, helping facilities deliver high-quality patient care and outstanding customer service. The top priority is that all data, from financial processing to clinical reports, is available online from anywhere in the enterprise.
Kindred is a healthcare services company that, through its subsidiaries, operates hospitals, nursing centers and a rehabilitation services business across the United States. As new companies are acquired, additional demands are placed on the IT infrastructure, such as supporting more users, company codes and ad-hoc reporting requests.
The IS department supports over 1,000 software applications, including an integrated financial, human resources and payroll system and business intelligence data warehouse. There are more than 1,300 servers connected by a robust frame relay network to 16,000 workstations supporting about 350 IS professionals.
Overall, Kindred’s employee base has risen to more than 56,000 employees spread across 138 active company codes. IT infrastructure has been responding to this growth, as evidenced by the database expanding 10-fold over eight years, from 200 gigabytes to 2 terabytes.
The company’s growth was making it difficult to complete month-end activities on time, leading it to upgrade its five-year-old server landscape with new high-performance servers. Subsequently the IS department ran a series of before-and-after benchmark tests and calculated the difference in system performance. We measured the time to run financial and HR reports, SAP transactions, overnight batch jobs, as well as online user response time. The upgrade provided a significant increase in speed, ranging from 24 percent to 800 percent, depending on the application and the number of users.
Unable to scale older machines along with the pace of the company’s expansion, the IS staff observed the strain on the system. It became more and more difficult to close the books within a 10 day window after month-end, an IT department service-level commitment. The database server experienced 100 percent utilization for two hours every morning during month-end close, as well as usage spikes that consumed all of the system capacity throughout the day. End-user online response time averaged above two seconds, with some processes experiencing severe delays. Night time batch and payroll runs were running into the morning hours and affecting online users.
Beyond addressing these performance issues, Kindred’s key objective was to find a solution that could accommodate new business and acquisitions, improve SAP application performance and increase uptime (high availability). Another requirement was a cost-effective solution, providing return on investment (ROI) in five years or less.
We sought the help of HP to architect a new scalable IT infrastructure that would support our SAP R/3 software and anticipated business growth. This meant increasing capacity, improving response time and increasing system availability. These objectives were met through two significant upgrades.
First, 15 legacy servers were replaced with nine HP Integrity servers (rx7640 and rx4640) with Intel Itanium processors, migrating from 32-bit systems to 64-bit to increase performance. These systems run 64-bit application software, including mySAP ERP 2005, Microsoft Windows Server 2003 operating system and Microsoft SQL Server 2000, which allows users to query, analyze and manipulate data over the Web.
Second, the servers were organized in clusters — pools of independent servers that work together as a single system. Server clusters can increase capacity and improve response time by allowing the software to spread the workload among multiple servers. This helps avoid having one server 100 percent utilized while other servers are underutilized. There are also highly available clusters consisting of two or more nodes that are exact mirror images of each other. If the primary system goes down due to hardware malfunction, fail-over software immediately makes its twin system the primary node.
With a project of this size, there are usually a number of decisions and implementation processes that have a significant impact on the outcome. One such decision was to contract HP Services engineers to rack and configure the servers before they were sent to us. They installed the OS Image and software we sent them. In addition, they configured IP addresses, system names, worldwide names for the SANs and other system parameters. Our alternative was to hire three or four external heads and probably take a several week schedule hit. Afterward, HP documented the configuration process and burned CDs of the server images.
We also decided not to wait for a server refresh that incorporated the latest dual-core Intel Itanium and the HP sx2000 chipset. These newer servers would have given us almost twice the performance of the single-core processor versions we implemented. Given our aggressive project schedule to remedy our seriously overtaxed infrastructure, such delays could not be accommodated.
By adding a keyboard, video and mouse (KVM) switch that allows multiple servers to share a console, system maintenance has become easier and more convenient. Administrators can remotely VPN into the network, connect to the switch and access the various servers.