Computer Assisted Healthcare, The Surmountable Insurmountable Challenge

December 29, 2011
Will it take a computer capable of reading a million books per second to help physicians review a patient's complete chart and appropriate, vetted knowledge in context?

Computer Assisted Healthcare, The Surmountable Insurmountable Challenge
Speech Recognition, Language Processing, and Reasoning Facilitation are Essential to Provide Adequate Baseline Care

For many years, the information overload challenge has been clear to us all.  With the advent of Meaningful Use requirements, like maintaining and evolving a useful codified problem list, we're finding that we don't have the time, much less the adequate definitions to keep up.  As I travel around the country to talk with physician executives, all of them see the challenge of moving to ICD-10 and related documentation improvements as an insurmountable challenge.  That is, insurmountable to do manually or with currently available technologies.

Last February, an AI-enabled computer system named “Watson” competed against two human Jeopardy! masters in the game and won.  To do so, it had to use natural language processing and reason over four terabytes of disk storage, including the full text of Wikipedia.  There's a nice summary of the hardware and software issues here:

Will it take a computer capable of reading a million books per second to help physicians review a patient's complete chart, the relevant medical literature, the quality and coding requirements, and assemble a note and orders with adequate reasoning to address our triple aim goal: better healthcare, better health, lower costs?  It might.

For a presentation that elaborates on the background and paints issues in detail, watch the video ("Jeopardy! Understanding, Questioning and Answering in Healthcare").

Jeopardy! Understanding, Questioning and Answering in Healthcare from Healthcare Informatics on Vimeo.

We have developed systems with the capacity to store and retrieve all the information we need to provide not just adequate, but quality baseline patient care.  My team, working in conjunction with users, and many others in our industry are now striving to evolve our systems to near the level of Watson, but without the astronomical cost associated with owning it.  Impossible?  I don’t think so.  After all, 20 years ago, the Internet as we know it today was considered by most a pipedream.

What do you think?

Joe Bormel, M.D., MPH
CMO & VP, QuadraMed

of 2Next


Good post Dr. Joe — informative and entertaining. Nice to see IBM just isn't sitting on their laurels. Must be a ton of dollars invested in Dr. Watson, probably almost as much as it took to get Dr. Joe to his level of expertise and capability.

For my viewpoint I think Watson can be helpful in medicine, but I do think that IBM still has a very long row to hoe. I am going to be most eager to see how they handle the 'art' side of medicine. If you think about all of the Jeopardy questions they tend to be object oriented, with black or white. The variability is in how you get there and how fast. This is where Watson and the contestants excel.

Now if the questions /answers included a greater amount of judgment I could get really excited. For lay people like me (non-MDs) there is a good book out today entitled "Your Medical Mind" by Jerome Groopman M.D. & Pamela Hartzband M.D. that highlights the issue of what happens when treatment decisions are conflicting and that much of the diagnosis /prognosis is judgmental.

I applaud IBMs effort, good to see some progress, but I would temper my excitement until they take on questions (medical or non-medical) fraught with at least a little judgment.

Happy New Year!
Frank Poggio
The Kelzon Grroup

Thanks for your comments, as well as your kind words about the post. I've received other feedback about the costs.

Companies spend billions on investments.

When an investment meets a market the size of healthcare, with the promise of recurring service revenue, the investment dollars are insignificant (for a large player or consortium of investors).

HP's recent acquisition on Palm of over a billion comes to mind as a rationalized yet failed investment. Pharma spent nearly an equivalent amount on Torcetrapib which also didn't pan out. These are risk/reward calculations; Watson investment dollars are not atypical.

At the same time, services like FaceBook (with three point eight billion in ad revenue this year) have invested probably a lot less. Investors will do quite well with the likely IPO that will emerge.

The Watson hardware, software and IP development is substantial but well within a moderate risk profile for an initiative with this kind of value. On the hardware side, Moore's law has been holding up from what I read.  Five years ago, the Sony Playstation Three hit the consumer market with nine independent processors.  From what I'm reading, Google, FaceBook, and Apple have all build data centers in North Carolina with enough cloud power to make Watson look pretty modest.

And most billion dollar health companies are exploring ways to franchise what they're creating. This technology will be necessary for accountable and affordable care, if only to drive down the administrative costs. This was pretty clearly shown in the slides underlying the video.

To your questions:

1. Deployment: There are several questions here. Technologically, this is exactly the problem that cloud-based services addresses. In terms of business models, IBM can either forge the business models quickly and win, or stumble and become irrelevant. Winners with names like Amazon books/on-line retailing in general and Apple's iTunes clearly won market share and survived with sustainable businesses. There is a long list of large and small companies that saw the same opportunity and failed. They either got the model wrong, or choose the wrong partners, or failed the marketing, or the simply didn't move fast enough. The requisite solution will be available and affordable across the market and will leverage available, interoperable EHRs. Independent of IBM's behavior, some vendors and provider organizations will deploy reasonable workflows that incorporate conversational dialogues facilitated by Watson (see Lynn Kosegi's comment above). Others will attempt more static models, academic solutions, or use Watson in a way that's more aimed at distraction than value creation.

2. Cost-Benefit and further development costs: You raise an interesting question. The answer comes from the following question - Will the episode-of-illness-treatment-costs (say, using ETGs) using Watson be greater or less than current actuarial costs compared to not using Watson? And, if there is a Watson-enabled advantage, how will that be distributed between payers, providers and consumers/patients?

What do you think?

Joe — I really enjoyed your Watson /Jeopardy application to medicine presentation! It was excellent!

I was struck on many levels that Watson is well suited to healthcare prediction as well as health prediction, using data, information, and knowledge that may be unstructured. The concern I would have is the blurring of exploratory and confirmatory analysis. There's a big difference between a series of case studies and a well constructed clinical trial. Do you see Watson sorting hypotheses by GRADE-ing evidence (Grading of Recommendations Assessment, Development and Evaluation) or similar authority strength methods?

Ann, Thanks for your comment. You are exactly right. HealthCare is rife with psycho-social and situational factors that factor prominently into decision making.

Take something relatively simple, like "patient refused the recommended therapy." Those words don't convey the half-dozen different cases that could represent:

1) the recommended therapy was inadequately communicated.
2) the recommended therapy was tried in the past and nearly killed the patient.
patient's "refusal" was more informative but the provider didn't have the time and sophistication to deal with it.  The provider dismissed the side-effect profile that the patient was communicating.
4) actually, the provider refused for the patient.
5) alternative, secondary recommendations were not considered or offered.
6) the rate of such refusals in this patient's practice are well beyond their peers.
7) ... the list could easily go on.

The original post and video provided a little background on "Toronto." Another minor mistake that I didn't include was what happened before Watson offered the wrong answer "What were the 1920s?" The prior contestant guessed exactly that and was told that the answer was wrong. Watson did not incorporate that new knowledge in realtime. If Watson had, at the least, it would have answered with it's second most confident answer/guess, not repeat the answer now known to be wrong.

So, to elaborate on your point, Ann, it may be necessary for these computer-assistance technologies to be conversational. They would need to incorporate solicited and unsolicited input through a dialogue.

Your other point is really even more profound. "Canned Algorithms" have very real limits, especially in the context of real-world patients, with their real contexts (co-morbidities, preferences, ability to participate in shared decision making, and pharmaco-genomics.) Today, we guess at these things at best in most cases. And those guesses factor in the experiences of the providers involved, as it did with the "throw in rabies" example in the video, where one doctor thought of it and the other didn't.

Fortunately, the Watson framework can simultaneously consider and evaluate hundreds of hypothetical considerations simultaneously, with its "best"ness-weighted decision rendered in three seconds or less, reasoned over millions of bits of graded (sic GRADE) evidence.

Historically, no single doctor-patient relationship in the past delivered that reliably. The promise far exceeds the limitation of a few canned algorithms. To your point, let's hope that the execution is well done, and with guiding principles of compassion and better care for patients.

Thanks for your comment and insight. The short answer is - yes and great point. At a minimum, Watson's scoring capabilities will need to be competent at evaluating strength of evidence. It will have the ability to do that in the context of situation, which, compared to current practice, could be a healthy improvement.

I recall during my MPH training that the statisticians and bio-epidemiologists had entirely different standards of goodness, although never in the same person! As elaborated in the presentation and the 700 chess masters analogy, Watson could offer a justification that would be better informed than the above average expert. If you're interested in this topic, there is at least one paper on the internet on Watson's wagering strategies that might provide a deeper insight on how it values authority.

Dr. Joe,

Interesting timing on your post. CMS just announced the 32 first "pioneer ACOs" 12/19. Like the IBM Watson approach, the CMS ACO attempts to shift the paradigm from one-off thinking (and accountability) to shared group thinking. Two approaches to harnessing the power of a network (IT or human) to produce a better outcome. The ACO approach is more about leveraging the knowledge already inside the heads of system members and achieving efficiencies by eliminating duplicative services and filling in inter-practice patient monitoring/compliance gaps. IBM/Watson focuses more on acquiring data that is available in patient databases and in knowledge repositories not already in the heads of the human system members.

Those 2 approaches are not mutually exclusive and, I believe, both will profoundly influence the organization of how medicine will be practiced by future generationis of clinicians.

Also must mention your blog comment on the pilots flying in the same plane as their"patients." The ACO theory addresses that in part, although not in any way that provider and patienst share physical outcomes: only by tying provider financial bonus incentives to the safe arrival of their patient.

I absolutely acknowledge the healthcare potential of Watson, and sincerely admire IBM for stepping into the arena. And I realize that adapting Watson for physician use is not exactly an overnight project. It will take considerable time.

But on a far more basic, and perhaps real world level, Watson is and will be almost astronomically expensive. According to the information I read on one of your links, the hardware alone cost IBM at least $3 million. I doubt the company is about to release the actual cost of the programming. IBM and those organizations that may provide content for a "Doc Watson" version will want to be reimbursed accordingly, and they will deserve proper remuneration.

Therefore, the first of my three questions for you, based upon your knowledge to date. Do you think that Watson, whenever the system is ready for prime time, will likely be available through some sort of hosted/cloud-type access so that the cost will be manageable for the average hospital?

Seriously, accepting for the sake of my question that the information Watson will provide will be superior, that won't matter if a doc in the average hospital or group practice can't afford to access it. And a follow-on question, if only the largest organizations with considerable resources will be able to afford to access the information, isn't it probable that most providers will be at a severe competitive disadvantage and this could have a negative effect of the overall availability of quality healthcare? I'm thinking accountable care.

Second, without standardization, it would appear that EMR interfaces will need to be developed. Is this rather expensive presumption on my part correct? If so, the vendor and consultant fees will undoubtedly add still another financial burden for providers. I think that would result in CFOs and CIOs being hard pressed by their boards to provide a cost-benefit justification, much less an ROI.

Please understand, I'm not trying to be contentious . . . for a change. I'm simply concerned that grassroots affordability has not been addressed in your post, which it probably shouldn't have been, but also it has not been addressed by the folks providing comments. I think this needs to be an integral part of the conversation. Otherwise, Watson could become akin to the average federal boondoggle that overlooks "how the heck are we going to pay for this?" The healthcare industry doesn't have the luxury to simply turn on the presses to print more money!

Thanks for an excellent post. It certainly provoked me to start thinking about this subject.



Thanks for your comments, and for your inspiring work referenced in the video in this blog.

A year ago, your work convinced me of the risks of accelerating down "blind alleys" without improving diagnostic accuracy. The resources section of the link you provided offers nice elaboration.

I do think that there is a lot of room in the "delivering better patient care" that's not a distraction. I think the "throw in rabies" example is on point. If Watson delivered Dr Mike's line to Dr Kate, and in his absence, a better diagnostic, therapeutic and productive encounter would have occurred for Julia.
As I referenced in the blog, unless the computer assembles the known documentation, orders, and problem list, and clarifies the ambiguities that make ICD-10 impossible to retrospectively apply to charts today, computers will simply fail to be adequately useful and adoptable. MU be damned. The impact of overwhelming amounts of information to an unaided human mind is completely predictable. Part of the solution is better characterizing the problem. Your work, and that of your collaborators, is extremely important.

Regarding your comment about the number of diagnoses to show, I think the situation may be better than you elaborated for readers. In my experience, some computerized inference systems can offer critical information beyond ranking. For example:

  1) Given the available information, your patient meets the classification criteria for Diagnosis:  XYZ (e.g. diabetes, TTP, etc), and ABC.

  2) Further, as was the case with QMR, the system could provide the incidence of the hypothesis diagnosis, and the evoking strength of strongest, most specific evidence. As we both know, these statistics are often highly location and presentation specific, which is why we need bayesian belief networks trained on the right data. It's beyond most humans to reason this way.

On a personal note, I have experienced the bias to not look for pulmonary hypertension in a friend who recently died from the fen-phen side effect this past June. I explicitly knew about the risk and didn't drive for the echo that confirmed the diagnosis. I just didn't want to find that, and used the rareness as an excuse to not look. The drug company's side effects monographs and providers involved also didn't use,  have access to, or publish the real probabilities. Watson would be indifferent, and unlike humans, be able to easily sort out stronger data from weaker. It could also better sort out wrong data.

What percentage of docs actually practice the Dempster-Shaffer theory of evidence (a technique to dismiss observations that are wrong) when necessary to reach correct diagnoses? I would guess the minority. Like me, we have a bias to lower our anxiety, sometimes at the expense of patient safety.

Thanks again, Mark.

Thanks Frank. Part of the feedback that I got when I presented this at Health TechNet was that the Wellpoint decision was brilliant for IBM.

The reasoning is that they need to make black and white decisions many times a day for treatment authorization decisions. As payers more closely partner with providers under shared programs (risk, savings, patient engagement), reasoning consistently and fairly over available information could become more important.

As I tried to call out, there are dimensions of the Health, Information and Technology puzzle that aren't going to be fixed by information appliances!

Interesting dialogue, here ( on why a computerized note may be overtly wrong.

  Original:  Why your EHR note may not be accurate by

The gist is that physician notes, especially when they contain portions that are computer generated content, can be wrong (or at least, less than right). This is compounded when a doctor edits the text. Now, the computer recorded depiction is internally inconsistent.

How is a human or a computer like Watson supposed reason over what are effectively the rantings of a mad man?

Thank you for your reply. I wasn't expecting such a detailed response!

If I understand what you're communicating, it appears that not only will the vendor, be it IBM or some third party, need a complex business model, the user organizations will also need the same to effectively define what their intended uses are for a healthcare Watson, and from them develop reasonable expectations. That makes sense, as does taking a cloud-centric approach.

As far as the cost-benefits, it appears we will need to take a wait and see approach for now until "Doc Watson" is unveiled. However, in my opinion, I can see consumers/patients becoming involved at no charge with Watson providing information to them much the same as WebMD. The benefit for the vendor would be derived from patients asking their providers, "You do use Watson, don't you?"

As always, thanks for helping to put our thought processes in gear . . . and traveling on the right path.


In the HCI on-line article,  12/20/2011, the article elaborated:

Cedars-Sinai, WellPoint to use IBM’s Watson


Cedars-Sinai's oncology team will help develop recommendations on appropriate clinical content for the WellPoint health care solutions. They will also assist in the evaluation and testing of the specific tools that WellPoint plans to develop for the oncology field utilizing IBM's Watson technology. The Cedars-Sinai cancer experts will enter hypothetical patient scenarios, evaluate the proposed treatment options generated by IBM Watson, and provide guidance on how to improve the content and utility of the treatment options provided to the physicians.

It looks like Watson-in-HealthCare is executing against the strategy that you described in the video.  It's interesting that description doesn't mention "treatment authorization."

Hi Joe,

I finally carved out time to enjoy your presentation - it was GREAT. The Watson team gave a presentation at our Diagnostic Error conference this year, and yours was highly complementary!

It will definitely be interesting to see how doctors respond to WATSON's suggestions in regard to diagnosis. WATSON can't function, like it did in JEOPARDY, giving only 1 answer, or sharing the top ranked three. It will have to give the top few (5? 10? 50 ?)

Others who have been working on this problem have really struggled with figuring out the best number to show (they settled on '1 page about 10), but there may not be any good answer to this question. Our minds certainly don't work that way!

I did a project with this other (ie other than WATSON) team a few years ago using 3rd year med students, and they were completely overwhelmed by all the possibilities that were presented. Experienced docs will do a little better, and maybe the extra suggestions will trigger something they hadn't thought of. The BIG question is whether this will lead to more correct diagnoses, or just more blind alleys !

Here's the link to our new Society:


Hi Joe,
I'm waiting to see how Watson factors in psycho-social and situational factors into its data.

Clearly computers can be a huge resource for clinicians determining diagnoses and reviewing best practices. But the idea that clinical decisions for complex co-morbid patients can be translated into canned algorithms seems to be the hopes of corporate healthcare and engineers.

Thanks for your comment.

Great doctors have a trick. When faced with a fat chart, they look for a recent note from another physician they know to be smart and compulsive. Then, they only need to go back that far in the chart to get a trustworthy synopsis until that point in time. Google and Watson today have the ability to weigh more reliable context from rantings using knowledge about the credibility of the sources.

Your point is a good one. Conflicting narratives exist today and have the potential to improve, worsen or both as more electronic documentation methods become prevalent. Humans and Watsons (sic) have and will deal with this dirty data.

I touched on this a bit here: . I pointed out that GPS navigation has become vital, and yet, it reasons incorrectly at times. That creates a completely new burden on the driver. As you point out, the same will certainly be true in the world of EHR notes.

I also pointed out here, that EMRs used by humans will contain mutually incompatible observations that can only be interpreted correctly by understanding where the speaker stands.  I vividly remember in my training when an infectious disease doc said that a patient's infiltrate must be tapped; a rheumatologist seeing the same patient on the same afternoon said that the infiltrate was most likely caused by a non-infectious condition and didn't need to be tapped.  In fact, it shouldn't be tapped.  Computers and electronic documentation wont make the world black and white.  And, with that, will come some degree of confusion.  Some false negatives.  Some false positives.  Having a good clinician will be as important as ever to sort through it all.

Thanks again for the observation.

Hello Dr. Joe: I love the fact that unlike some conversations around the use of health information technology, you do not leave the PHYSICIAN and the need for compassionate and high-quality care out of the picture. So many conversations around the use of technology in healthcare target analytics and reporting for meaningful use and other reporting requirements and don't focus on the need to improve documentation at the source in support of care of the patient!

The use of technology for computer assisted coding is an interesting conversation. Many claims abound about accuracy, precision, improved coder productivity - but they miss a big point. If the documentation isn't there to begin with (and for ICD-10, it won't be) then computer assisted coding technology isn't much help.

Watson is also intriguing, but your point about Watson not incorporating the new knowledge in realtime is an excellent one. The talk about Watson as though it is a plug-and-play, one-size-fits-all, solution for all our healthcare problems is a gross over-simplification. To quote Detlef Koll (whom you know), "HIT isn't a refrigerator. You can't just plug it in and let it run."

My colleagues at M*Modal and I agree with you wholeheartedly that these types of technologies must be conversational. There must be a dialogue in place with the physician and other consumers of health information that helps to improve documentation at point-of-capture. These dialogues must open the way for the technology to learn from the human expert users. The technology can never and will never replace human beings. But can it assist the human expert? Absolutely - IF the technology becomes an integral and logical part of the workflow, if it becomes an intrinsic part of otherwise disconnected processes and systems, (the last thing any healthcare provider needs is yet another system that doesn't talk to any of the other systems in use) and if - as you say - "the execution is well done, and with guiding principles of compassion and better care for patients." I love that Dr. Joe! Well said.

Happy holidays and I look forward to talking with you in the new year!

Lynn Kosegi
  Director Health Information Services


Thanks for your kind words.

I like the refrigerator metaphor. It's telling. A related metaphor that captures a related aspect is "A hammer is not a house." People want to buy HIT and then have not only a house, they have a vision for a warm, well-stocked loving home.

Turning on HIT should be the beginning of the project of building a home, not anything less. Yet it often is.  And, to your point, there isn't enough information captured today to adequately classify patients to promote better care in too many instances.

Thanks for your perspective.

Interested readers will find value in Lynn's recent presentation at AHIMA on NLP/The Magic Bullets, here for your convenience:  /Media/BlogReplies/2011-10 AHIMA_2011_MModal_FINAL ppt Compatibility Mode_0.pdf
It has been an inspiration for me since I saw Kosegi and Fritsch present it two months ago.