Big decisions - Certification and Evaluation? | [node:field-byline] | Healthcare Blogs Skip to content Skip to navigation

Big decisions - Certification and Evaluation?

March 12, 2009
by Joe Bormel
| Reprints

Big Decisions - Certification and Evaluation?

Do meaningful use, qualified provider, and certification hit the mark?

David Classen Revisited

Thank you

Dave Raths, for the "Big decisions await HHS secretary" post yesterday. Dave did a great job of capturing the essence of the HIMSS Webinar. He ended his post with a series of questions that invited more than a comment. Although I strongly agree with Mark Leavitt's conclusions, that Certification was created to meet a real need that was previously missing in our national strategy, and that it has earned credibility, I think one more topic needs to be in the conversation. Evaluation of live products, especially those addressing CPOE.

This is not a new idea.

Fran Turisco does the best job on these blogs of laying out the problem and solution here. Part of the gist is that even certified EMRs with CPOE, capable of catching and preventing lethal mistakes, can be configured and implemented such that they get failing scores.

There's an excellent review of the issues and options in my prior blog post, "

When Rules Collide, Alerts Fatigue, and Disruptive Innovation may be necessary." From that post, here is a small piece (the original with its comments are worthy to scan on their other dimensions):

David Classen’s 2008 HIMSS presentation entitled, "Evaluation of Implemented EHRs," looked at this issue in detail, in the real world. Deployed EMRs were tested using the Leapfrog flight simulator.

CDSS scores dropped to the 10% range (50% is passing), as a result of deferred or deliberately disabled CDDS.

These systems (from many major vendors) had CDSS capable of meeting or exceeding the 50% passing grade threshold; the CDSS was turned off, reducing or eliminating the safety benefit. EMRs often performed at the 10% level for decision support.

Classen has a distinguished career, having established himself as an expert in Adverse Drug Events two decades ago, and subsequently contributing to the topic from every major perspective. Per Mark's use of the word, "credible" and HIMSS' call for "market tested" approaches, the question of the evaluation David Classen, Jane Metzger, and Fran Turisco describe should be explicitly on the table.

If chief of staff

Rahm Emmanuel is right that “you never want a serious crisis to go to waste,” and if the goals are to expand coverage AND control costs, aren't certification and evaluation both critical?



How refreshing! Of course evaluation should be part of the process. Or perhaps, I've understated its importance. Evaluation must be part of the process.

What good is certification that all the bells and whistles that should be in a system are actually in the it if that same system doesn't work in the real world? The idea that certification assures performance is utterly ridiculous. I do believe this would be called "snake oil" in the Old West.

Independent evaluation of systems would add a significant level of assurance that a particular EMR can meet expectations... for implementation by a physician group through systems designed for major healthcare organizations. Naturally, there's a certain reality factor that comes into play, because system implementations, particularly large ones, are highly customized. Therefore, certain standards would need to be established that would validate the evaluations. But this is not necessarily a bad thing.

Such standards could help to bring system expectations in line with the capability and functionality of existing technology. System expectations very often run way too high, vendors over-promise, and even references can be flawed. After all, what hospital, for instance, wants to admit it made a multi-million dollar mistake, especially when its vendor has "promised" upgrades on a specific schedule... that are seldom on time? It's not as easy for CIOs to "bury" their mistakes as it is for, well, you know the old saying. No offense meant.

I think we all accept the fact that at virtually every level, implementing an EMR is a "career" decision. For physician practices, installing the wrong system could lead to bankruptcy. And in a hospital environment, the wrong system could be career-ending for at least the CIO and CFO. The more hard info all the decision makers can have available, the better. And absolutely nothing beats system evaluation.

Windows releases receive more independent scrutiny than EMR systems, and they certainly don't have any impact on a par with life or death. System evaluation would certainly move out the target year when they become mandatory, but the front end savings should be obvious to providers and the federal government. After all, it wouldn't take a rocket scientist to count the number of times the feds have tried to implement systems that never worked. And for healthcare providers that mistakenly buy the wrong system, there's no second grant from the feds to try covering errors in judgment. Welcome to the hard realities of the real world.

Although I haven't really addressed the savings EMRs may (or may not) generate in the long term, I could not agree more with Anthony's comment. That's reality, too!


It seems most of these overestimations of savings come from the mistaken notion that implemeting these systems (in terms of fostering effective use, not just implementation in the technical sense) is MUCH easier than it actually is.

Thanks Anthony and Jack for the validation.  I'm glad to see some compelling leadership that benefits, like savings, don't just happen.  They need to be managed to.

Jack, you made several compelling observations that I didn't miss, especially the reference to career risk, burying mistakes, and the sobering examples from popular operating systems.  Implementation is, to quote David Brailler, a battlefield; it's often a war of attrition as well, for those trying to bring about improvement.

Since my original post, there have been some noteworthy reactions:

1) The WSJ  forums on the 80B$ exaggeration is here.

2) Bates, Halamka, and Middleton wrote a response as well.  Here's an excerpt:

"We already have clear evidence demonstrating that electronic records improve care and reduce costs when implemented well in specific settings.  Three of many examples include:

1.   We (Middleton) have published a detailed case study of the cost and quality benefits of EHR at Family Care of Concord, NH demonstrating net benefits per clinician per year of $30,324. This is comparable to the cost savings we've estimated in our academic health centers.

2.   We (Middleton, Bates) have published a cost-benefit analysis of  hospital-based provider order entry documenting at net savings of $1.7 million per year from drug dosing guidance, nursing time utilization, and error prevention.

3.  We (Halamka, Bates) have documented cost savings from automating radiology and medication ordering processes to significantly reduce utilization, pharmacy costs, and staffing.

I believe their data and interpretation.  Each has an independent, multi-year track record for publicly presenting their work and addressing the scrutiny.  Each has the experience that comes from implementing a solution; this, of course changes our understanding of the problem in highly valuable ways.

They've also been very candid about the risks extrapolating results nationally.  That always requires modeling and assumptions.  That makes everyone nervous about the power of the conclusion.

This is exactly why evaluation, based on clearly stated requirements and use value cases need to underlie the savings claims.  As I've covered previously in this blog, hereDoug Thompson here, and the previously posted comments about Peter Orszag's Health Warning.

Two examples that give me pause:

- Both my dad and mother-in-law had expensive procedures repeated within days of original tests, looking for carotid artery disease.  Doctors had access to the results, although perhaps not electronic.  They choose to repeat the tests for reasons that wouldn't change by adding an EMR.  So, I don't see, in my world, a likely 10% reduction in duplicated or unnecessarily repeated tests of this type.  Is that a small or large component of the expected savings?  When I've seen dispassionate third parties present local, data-informed projections to hospital CFOs, the savings projections are usually dismissed.  We've all experienced this, even when benefits are heavily discounted for the usual reasons.

- One of my clients experienced a cost savings by reducing the ordering rate for a test called ESR, aka a sedimentation rate.   I find it highly unlikely that the result would occur outside of a teaching hospital.  I don't see that this kind of savings would have impact on FTE counts at 99% of provider organizations, nor impact supply costs.  I know from my ACPE training, that ordering rates, by themselves, are a very dangerous proxy to use for total financial impact of a management intervention.

Apparently, we're going to need to revisit this topic, at least annually. Here's the latest publication. My take away? Of course evaluation will be a critical for reality-based participants. Read on:

Mixed Results In The Safety Performance Of Computerized Physician Order Entry

Jane Metzger1,*, Emily Welebob2, David W. Bates3, Stuart Lipsitz4 and David C. Classen5

1 Jane Metzger ( is a principal researcher at CSC Healthcare in Waltham, Massachusetts.
2 Emily Welebob is an independent consultant in Indianapolis, Indiana.
3 David W. Bates is division chief for general internal medicine at Brigham and Women's Hospital in Boston, Massachusetts.
4 Stuart Lipsitz is a researcher at Brigham and Women's Hospital.
5 David C. Classen is an associate professor of medicine at the University of Utah in Salt Lake City, and is also with CSC Healthcare.

Computerized physician order entry is a required feature for hospitals seeking to demonstrate meaningful use of electronic medical record systems and qualify for federal financial incentives. A national sample of sixty-two hospitals voluntarily used a simulation tool designed to assess how well safety decision support worked when applied to medication orders in computerized order entry. The simulation detected only 53 percent of the medication orders that would have resulted in fatalities and 10—82 percent of the test orders that would have caused serious adverse drug events. It is important to ascertain whether actual implementations of computerized physician order entry are achieving goals such as improved patient safety.

Roughly the same time I wrote this, Jerome Groopman and Pamela Hartzband wrote an opinion piece in WSJ March 11, 2009 "Obama's $80 Billion Exaggeration", available here.

It elegantly captures the same sentiment of sobriety articulated by pretty much all of the seasoned adventurers who read or write the HCI blogs:

[Obama] EMRs ... would save some $80 billion a year, safeguard against medical errors, reduce malpractice lawsuits, and greatly facilitate both preventive care and ongoing therapy of the chronically ill.

Following his announcement, we spoke with fellow physicians at the Harvard teaching hospitals, where electronic medical records have been in use for years. All of us were dumbfounded, wondering how such dramatic claims of cost-saving and quality improvement could be true.

... [the real cost drivers] ...

All agree skyrocketing health-care costs are a dangerous weight on the economic welfare of the nation. Much of the growing expense is due to the proliferation of new technology and costly treatments. Significant monies are spent for administrative overhead related to insurance billing and payments. The burden of the uninsured who use emergency rooms as their primary care providers, and extensive utilization of intensive care units at the end of life, further escalate costs.

Another, even more succinct way to make this is in my Vowel's post, here.  If those Vowels and the sequence are relevant, Groopman and Hartzband's argument is that the administration at best focuses on Administrative Costs, and the skips the considerably more important issues, where the $80 billion annual saving opportunity truly exists.  Addressing only the administrative benefits of EMRs completely misses the upward sloped trajectory of costs.  I encourage everyone to read the full Groopman and Hartzband article.

This comment is related to evaluation, specifically as articulated in this original post.  Evaluation is specifically about the difference between putting in an EMR and putting in an EMR and deriving a reasonable portion of the benefits that both Obama and Groopman / Hartzband articulate.  Without explicitly defining the Vowel spectrum of benefits, defining meaningful use and certification requirements will have the degree of goal vagueness and benefit translation challenge that bothers Groopman and Hartzband.  And me, too.