Skip to content Skip to navigation

When Rules Collide, Alerts Fatigue, and Disruptive Innovation may be necessary

Printer-friendly version

In May 2008, Josh Peterson, MD, MPH, published a perspective entitled: "Integrating Multiple Medication Decision Support Systems: How Will We Make It All Work?"Â It's a beautifully balanced piece, describing the problem and prescribing proven, actionable advice.

Why This Blog? (clearing up a common misunderstanding, before we go on)

A blog is not a place to elaborately review or even summarize beyond a very high level. A blog is a perfect place to share observations worthy of discussion and have a conversation with interested, thoughtful and passionate readers.

The Challenge of Mainstream Models For Integrating CDSS

1. Portions of CDSS systems are often turned off, because of information "noise" and process [alert] "fatigue". We are all familiar with this problem and the common response, deferring and/or disabling CDDS.

2. David Classen's 2008 HIMSS presentation entitled, 'Evaluation of Implemented EHRs,' looked at this issue in detail, in the real world. Deployed EMR's were tested using the Leapfrog flight simulator. CDSS scores dropped to the 10% range (50% is passing), as a result of deferred or deliberately disabled CDDS. These systems (from many major vendors) had CDSS capable of meeting and exceeding the 50% passing grade threshold; the CDSS was turned off, reducing or eliminating the safety benefit.

The Need for Leadership

Dr. Peterson makes a few statements that speak specifically to CIO and CMIO leadership, and offers an ominous observation regarding how this is going in the USA.

- "The creation of a high-quality medication CDSS may take more time, evaluation, and refinement than initially has been envisioned." (expectation setting leadership)

- "Unfortunately, at this writing, such an infrastructure is an underfunded and distant goal in the United States, while other countries such as the United Kingdom and the Netherlands are making tremendous progress." (influence leadership)

I strongly agree. Are their pragmatic, available options? Let's kick it around.

The Current State

With or without analytic CDSS, EMRs objectively create value. They reduce and/or eliminate delays and errors in many care processes. The gains possible and proven with CDSS are much larger.

Most publicly discussed efforts to address the "turning off the CDSS alerts" problem involve a combination of
 1) adding provider-specific preferences around specific alerts and classes of alerts;Â
 2) adding more specific dose checking, dose recommending, and dosing wizards/dialogues; and
 3) adding specific screens, templates, and wrap-around systems that handle specific contexts very well.

All of those things are needed at this point. But did you notice the "adding", "adding", "adding" language? (I tried to make it easy for you.)

Does This Call for Disruptive Innovation?

I had the privilege, yesterday, to attend a presentation by Marion J Ball, Ed.D, whom most of you know, is an IBM research fellow, Hopkins Professor, Member of the IOM, Fellow of ACMI, HIMSS, CHIME, and member of HCI's Editorial Board. Â

She hit on the "adding, adding, adding" problem, in describing Disruptive Innovation in the talk. She defined DI by referencing Clayton Christensen's definition "... a technology that brings a much more affordable and accessible product or service that is simpler to use into the market." Â

Things don't get simpler by continually adding to them. And disruptive innovation is characterized not by adding or removing from the existing paradigm, but changing it and simplifying it. (e.g. Mainframe to PC, or rotary phone to iPhone, through several successive innovations.)

To frame it differently she shared the following quote which I'm sure resonates with all of us:

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.
    - Antoine de Saint-Exupery (1900-1944)


We have CDSS. We routinely disable it, after we deliberately selected these systems to improve safety through CDSS.

Within CDSS, we have rules and drug checking (drug-drug, drug allergy, duplication, dose, formulary, substitution, geriatric, renal, and other checking and guiding). There are other forms of CDSS.

This CDSS reasons over order intention (unsigned order scratchpads), order sets, pathways, plans, and clinical documentation, like medication reconciliation documents. Â

We're adding in more layers, as Dr Peterson and I have elaborated here. Not surprisingly, CDSS reliability is decreasing predictably and getting more complex, as the number of parts increase.

To borrow from Dr. Peterson, '

[we need a] process of discovery and refinement that ... lead to a standardized "cockpit" for knowledge management in clinical settings.


In May 2008, Josh Peterson, MD, MPH, published a perspective entitled: "Integrating Multiple Medication Decision Support Systems: How Will We Make It



I just received a kind note of praise for this blog post from Jerry Osheroff. Jerry and his co-authors created a wonderful workbook, addressing very pragmatically the "so what can and should you do" to improve care through clinical decision support, at your organization. It's in it's 5th revision, or 100th, if you count the drafts over the last few years!

Jerry says:

This month we'll be shipping off our new med mgt CDS guide to HIMSS for publication later this year ( We don't have any magic solutions, but do think we've provided some thoughtful considerations from a broad cross section of stakeholders that is very much in line with the sorts of issues you and Peterson raise.

I have read the CDS guide and strongly recommend any one interested in this area becoming familiar with it.

Anthony, Thanks for your kind words and the pointers to Jim Feldbaum's work. I am now caught up.

I thought Jim's guidance about alert fatigue was prudent and in line with both Dr. Peterson's article, as well as a CDS for CPOE consensus conference I attended in 2005 (led by Dr. Gerard Burns, then at Hackensack, and covering lessons learned by clients of nearly every major vendor and the home grown systems.) Technology development and implementation is still lagging in the industry, from what I'm seeing.

As alluded to in my piece above, and called out by Jim, we're still struggling with making sure alerts (or guidances) are relevant. For the most part, we're pushing messages to the provider, warts and all, and asking those providers to figure it out. It's profoundly disrespectful, in addition to being fatiguing and unsafe.

As we move forward, CDSS will be generating more and more internally inconsistent recommendations, unless we dramatically up our HCIT game.

For example, if one rule says 'the patient needs to be on a beta blocker for their heart disease,' while another rule says 'never give the patient a beta blocker because of their lung disease.' Should the computer provide both recommendations to the clinician and tell them to figure it out? Is the patient safer as a result?

Add in diabetes, depression and a diuretic. Do those earlier rules have the same relevance?

On the topic of communication 'hand-offs', the re-engineering work should include turning them into 'handshakes.' That's a reference to several of Jim's posts with flawed hand-offs, including the personal horror story.

The 'handshake', IMHO, should include 'checklist' technology. Atul Gawande and Peter Provonost did us all a great service in clarifying the value and role of checklists (as well as their relative absence in healthcare). Someday, I'll learn how to embed links in my comments until then, the checklist article is at:

This is a very timely subject for me. Thanks for the postings all. My current project includes meds orders and meds admin by Doc's, PA's, Nurses, and the Pharm D's receiving and reviewing orders. Now you all have me good and warned that this will be no small task. Oh well, here we go.

I have had preliminary interviews with most of the providers in order to establish the scope of the meds subsystem in the E.D. and Urgent Care settings where we will go-live first. I also had first meetings with the Pharmacy people and the Pharmacy vendor. I discovered differing expectations. Clarifying scope and basic workflow had not been accomplished prior to contract.

I have read the articles referenced and I thank you for those. I am listening and would love to hear from others on this topic. I am counseling for a reasonable approach with my clients, but I'm not sure I know what reasonable is now.

Thanks for the synopsis Dr Joe. I will find a way to be sure those three bullet points make their way into the minutes of a Medical STEG (the TEG stands for Transformation Expert Group) meeting along with the guidance regarding measurement and management, high level as well. These are Docs, so the references will be absolutely necessary.

Hi Joe,
I am very much enjoying this rich discussion thread. At Partners, we continue to struggle with creating a high-utility CDSS context for clinicians. We view the sweet-spot as "Important to remember but easy to forget" which lines up nicely with many of Josh Peterson's points about going after errors of commission or omission that occur frequently and are likely to cause significant harm. We have found that the three biggest factors making it difficult to provide high-utility are:

1) the technical architecture makes it difficult perform the kind of sophisticated, integrated inferencing that would avoid multiple alerts firing for any given order. For example, for a proposed drug order we might check a variety of considerations including drug-drug, drug-allergy, drug-diagnosis contraindication, drug-alternative lower cost available, drug-age, drug renal function, drug-liver function. Currently these kinds of inferences fire in multiple places so it is technically difficult to manage at the "meta-rule" level to provide only high-utility feedback. We tier alerts into 3 levels of risk, which does help. We track alert overides and retire or demote alerts that are low utility. We are currently on a path to migrate to a more flexible technical architecture.

2) A second major barrier is not having adequate context to be "anticipatory" enough to surface good choices to the clinicians before they propose the sub-optimal ones, avoiding interruptions. Even if we had a great architecture in place, without a rich context of patient state and provider context, it can be difficult to avoid notifying an oncologist that a patient has "expectedly" abnormal lab results

3) And finally, if we had great architecture and access to very rich context, we still have the big problem of knowledge acquisition. Even without rich context and architecture, it's a big job to maintain and update even rules of low complexity. Broadening the CDSS content so that it flexes by context such clinician specialty or patient co-morbidities makes it a bigger job. Genetics and molecular diagnostics will really up the ante.

Our team thinks about and works on solving these problems in a scalable way every day, it's costly to solve these problems so progress is at more of a turtle's pace than we would like.

Great post Joe as an additional discussion point, I'd like to provide the link to an article we published by fellow blogger Jim Feldbaum on alerts and, more specifically, how to avoid alert fatigue. Have you read this Joe? If not, I'd be very interested to see what you think. I've already sent Jim the link to your post, so hopefully he will weigh in here. Jim posted one of my favorite blogs of all time (harrowing to say the least) Bad Medicine, a Personal Horror Story

David, HCI does provide me notifications of posts, so, my timely response reflects this and nothing else!

I wanted to share an extremely high level synopsis, true for 2008:

1) You must provide highest-order drug-allergy and drug-drug alerts. Your content provider needs to flag those for you.

2) You should follow Jim's and Josh's guidance regarding measure and management they're excellent. Both have costs (the right resources and time). Investment decisions are local and harsh.

3) Clinicians must be made aware that, by design, the computer will not always protect them. Their vigilance needs to be higher, not lower, than pre-go-live. It is a well described phenomena that errors go up, and, that there are unintended consequences post go-live.

Be brave, smart, and keep going!