Skip to content Skip to navigation

As CMS Updates its Quality Ratings for Hospitals, Experts Question Their Validity

August 2, 2016
| Reprints

Last week, the Centers for Medicare & Medicaid Services (CMS) published the first release of its overall hospital quality star ratings that the government says reflects comprehensive quality information about the care provided at U.S. hospitals.

Indeed, the new overall hospital quality star rating methodology takes 64 existing quality measures already reported on the agency’s Hospital Compare website and summarizes them into a unified rating of one to five stars. As written by Kate Goodrich, M.D., director of center for clinical standards and quality at CMS, at the time of the release of the ratings, “The rating includes quality measures for routine care that the average individual receives, such as care received when being treated for heart attacks and pneumonia, to quality measures that focus on hospital-acquired infections, such as catheter-associated urinary tract infections. Specialized and cutting edge care that certain hospitals provide such as specialized cancer care, are not reflected in these quality ratings.”

Earlier this year, the American Hospital Association (AHA) and other industry stakeholders wrote a letter to CMS pushing the agency to delay the release of the new ratings system, originally scheduled for April. The AHA has long opposed the star ratings system, arguing that it doesn’t display an accurate portrayal of quality or patient experience. As such, CMS said it listened to stakeholder feedback, delaying the release of the ratings until now “to give hospitals additional time to better understand our methodology and data.”

Despite the delay, in a July 27 statement, AHA said, “We are further disappointed that CMS moved forward with release of its star ratings, which clearly are not ready for prime time. As written, they fall short of meeting principles that the AHA has embraced for quality report cards and rating systems. We want to work with CMS and the Congress to fix the hospital star ratings so that it is helpful and useful to both patients and the hospitals that treat them.”

Similarly, the Association of American Medical Colleges (AAMC) also released a statement criticizing the quality ratings, attesting that “They are based on a deeply flawed methodology that does not take into account important differences in the patient populations and the complexity of conditions that teaching hospitals treat.” The organization further took issue that facilities are being measured on an equal basis, which should not be the case at all since that’s not the reality that hospitals face.

What’s more, a recent Kaiser Health News report dug further into the ratings, revealing that just 102 of the more than 3,600 hospitals that CMS rated received the top rating of five stars. If that doesn’t jump out to you, maybe this does: Medicare gave its below average score of two-star ratings to 707 hospitals, including some of the most prominent and well-renowned organizations in the country such as Geisinger Medical Center in Danville, Pa., Beth Israel Medical Center in New York City, Tufts Medical Center in Boston, MedStar Washington Hospital Center in D.C., and many others, according to Kaiser Health News.

Due to these bizarre results, I dug deeper into Medicare’s ratings, which include 64 of the more than 100 measures reported on Hospital Compare, divided into seven measure groups or categories: mortality, safety of care, readmission, patient experience, effectiveness of care, timeliness of care, and efficient use of medical imaging. CMS collects the information on these measures through the Hospital Inpatient Quality Reporting (IQR) Program and Hospital Outpatient Quality Reporting (OQR) Program, noting that “a hospital’s overall rating is calculated using only those measures for which data are available.”

According to CMS, for each hospital, a hospital summary score is calculated by taking the weighted average of the hospital’s scores for each measure group or category. The table below shows the weight applied to each measure category. The hospital summary score is then used to calculate the overall rating.

Source: CMS

To this end, AAMC said that CMS used more than 60 measures to calculate ratings for teaching hospitals and as few as nine measures on some hospitals that treat patients with less complex conditions or that treat a limited number of conditions. AAMC also said that its analysis of the ratings “has confirmed that the lower the number of measures a hospital reported, the more likely a hospital was to receive a higher star rating. In fact, hospitals that reported on only 60 percent of the metrics or less received almost half of the five-star ratings.”

In the aforementioned Kaiser Health News report, Elizabeth Mort, M.D., chief quality officer at Massachusetts General Hospital in Boston, which received four stars from CMS, was quoted as saying, “I don’t put any credence in this. Don’t clutter it up with measures that have no place being there,” such as infection and readmission measures that she said were not well designed to compare dissimilar hospitals.