Skip to content Skip to navigation

Cautionary Tale: How to Read (and Not Read) a Study Dotted with Asterisks

March 25, 2014
| Reprints
Why one JAMA article analyzing PCMH success needs to be handled with caution


One of the more interesting moments, for me, at the HIMSS Conference last month in Orlando, came during the impromptu press briefing that Karen DeSalvo, M.D., the National Coordinator for Health IT gave, on Thursday, Feb. 27. It was her first, and Dr. DeSalvo appeared slightly nervous; but she handled the grilling from an eager-to-grill press corps with aplomb.

The first question, though, came absolutely out of left field. One of the 20-some journalists gathered at the press briefing gave a very long-winded introduction to her question, and then essentially challenged Dr. DeSalvo to justify or explain the findings of a study published just the day before in the Journal of the American Medical Association. Dr. DeSalvo very politely responded that, of course, she hadn’t had time to read or review the JAMA article yet, but did address the broader implied question.

Here was the question: “Given the results of the JAMA study, Dr. DeSalvo, would you say that the patient-centered medical home [PCMH] model should be considered a failure?” Okay, then. Well, let’s plunge into the JAMA article and consider why the question was truly absurd, and what the implications are when people rush to judgment based on one narrowly focused study of specific aspects of a very broad, complex phenomenon—and then I’ll share what Dr. DeSalvo said in response to the question.

The article, designated as an “Original Investigation,” bore the title, “Association Between Participation in a Multipayer Home Intervention and Changes in Quality, Utilization, and Costs of Care,” and was written by Mark W. Friedberg, M.D., MPP; Eric C. Schneider, M.D., MSc; Meredith B. Rosenthal, Ph.D.; Kevin G. Volpp, M.D., Ph.D.; and Rachel M. Werner, M.D., Ph.D. Those authors performed a very detailed statistical study, with the objective being “To measure associations between participation in the Southeastern Pennsylvania Chronic Care Initiative, one of the earliest and largest multi-payer medical home pilots conducted in the United States, and changes in the quality, utilization, and costs of care.” (The study was sponsored and funded by the Commonwealth Fund and Aetna.)

Essentially, what Dr. Friedberg and his colleagues did was to identify and recruit participant leaders from 32 primary care practices participating in the Pennsylvania Chronic Care Initiative (PCCI), a statewide multi-payer medical  home pilot, and compare their outcomes data around changes in quality, utilization, and costs, with data from 29 practices not participating in the PCCI program. Among the 11 clinical outcomes studied were changes in hemoglobin a1c testing, changes in normal/abnormal hemoglobin a1c status, and the execution of nephropathy monitoring, breast cancer screening, and colorectal cancer screening.  Hospitalizations, emergency department (ED) visits, and ambulatory care visits were analyzed with regard to utilization. And “propensity-weighted and –adjusted costs and utilization” were analyzed for differences between patients in the two different groups, after three years of participation in the PCCI program, on the part of the participating medical groups.

So what happened? Broadly speaking, there was very little positive difference in outcomes or in cost reduction among patients cared for by the pilot-participating medical groups, versus those cared for by non-participating medical groups. Indeed, some patients cared for by non-participating medical groups actually fared slightly better in some of the outcomes categories.

The authors’ conclusion? “Despite widespread enthusiasm for the medical home concept, few peer-reviewed publications have found that transforming primary care practices into medical homes (as defined by common recognition tools and in typical practice settings) produces measurable improvements in the quality and efficiency of care. The southeast region of the PACCI, which featured relatively generous financial support from six commercial and Medicaid health plans, is to our knowledge the first multi-payer pilot in the nation to report results over a three-year period of transformation,” the authors wrote. “We found that practices participating in the PACCI pilot adopted new structural capabilities and received NCQA certification as medical homes. Our evaluation also suggests,” they added, “that the quality of diabetes care improved, but we found few statistically significant results and no robust associations with utilization or costs.”

And now come all the asterisks. First, the study’s authors freely admit that they studied primary care practices whose physicians, while participating in the PCCI pilot program, were not enmeshed in programs sponsored by large integrated health systems. These are and were smaller primary care practices. Second, as the authors note, “This pilot—the first of the PACCI regions—was focused on quality improvement for chronic conditions and featured early financial rewards for NCQA recognition, possibly distracting from other activities intended to improve the quality and efficiency of care. In subsequent regions,” the authors noted, “PACCI organizers placed less emphasis on early NCQA recognition so that practices could focus more fully on learning collaborative participation.”