IBM Watson’s Shortcomings Investigated in STAT Report | Healthcare Informatics Magazine | Health IT | Information Technology Skip to content Skip to navigation

IBM Watson’s Shortcomings Investigated in STAT Report

September 7, 2017
by Rajiv Leventhal
| Reprints

An investigation into IBM Watson by STAT News has revealed that the artificial intelligence supercomputer has not lived up to its potential.

The piece, penned by Casey Ross and Ike Swetlitz, examined Watson for Oncology’s use, marketing, and performance in hospitals across the world, from South Korea to Slovakia to South Florida. The in-depth article included interviews with doctors that have deployed Watson at prominent healthcare institutions as well as other healthcare experts—many of whom seem far from thrilled about the results Watson has delivered so far.  

The idea behind Watson for Oncology is to help physicians quickly identify key information in a patient’s medical record, surface relevant articles and explore treatment options to reduce unwanted variation of care and give time back to their patients, according to the website of Watson Health, a unit of IBM that was launched at the 2015 HIMSS conference and which employs more than 7,000 people.

But according to the STAT article, Watson for Oncology’s “treatment recommendations are not based on its own insights from these data. Instead, they are based exclusively on training by human overseers, who laboriously feed Watson information about how patients with specific characteristics should be treated.” In fact, investigators said that while IBM sales executives say that Watson for Oncology possesses the ability to identify new approaches to cancer center, in reality, “the system doesn’t create new knowledge and is artificially intelligent only in the most rudimentary sense of the term.”

Despite being supported with major funding and incredible brand marketing, Watson is falling short in its ability to transform cancer care, sources in the article said. According to the STAT piece, “Three years after IBM began selling Watson to recommend the best cancer treatments to doctors around the world, a STAT investigation has found that the supercomputer isn’t living up to the lofty expectations IBM created for it. It is still struggling with the basic step of learning about different forms of cancer. Only a few dozen hospitals have adopted the system, which is a long way from IBM’s goal of establishing dominance in a multibillion-dollar market. And at foreign hospitals, physicians complained its advice is biased toward American patients and methods of care.”

What’s more the authors attested, “The actual capabilities of Watson for Oncology are not well-understood by the public, and even by some of the hospitals that use it. It’s taken nearly six years of painstaking work by data engineers and doctors to train Watson in just seven types of cancer, and keep the system updated with the latest knowledge.”

Watson, the Jeopardy!-playing supercomputer, is built to mirror the human learning process through the power of cognition. Its executives, some who were interviewed for the STAT piece, attest that Watson is till on an upward trajectory and is a major part of IBM’s business. They admitted that Watson is still in its infancy but that it’s growing rapidly; by the end of this year, noted the executives, “The system will offer guidance about treatment for 12 cancers that account for 80 percent of the world’s cases.”

Watson has had both successes and failures reported in the media in recent months. A Forbes report released the day before IBM’s global CEO Ginni Rometty's keynote at HIMSS17 revealed that the partnership between IBM and MD Anderson Cancer Center, part of the University of Texas, soured to the point where the $62 million project for the cancer center to deploy Watson had been scratched.

Bizarrely, the Forbes report also noted that MD Anderson is still on the hook to pay for the whole thing and that its leader in charge of the project never got approval for the implementation from the center's IT department. Questions remain regarding MD Anderson's current relationship with IBM,

Lynda Chin, M.D., who oversaw the Watson project at MD Anderson before it fell apart, told STAT reporters that it was quite challenging to make the technology functional in healthcare. “Teaching a machine to read a record is a lot harder than anyone thought,” she told STAT, noting how her team spent countless hours trying to get the machine to deal with the idiosyncrasies of medical records.

To the contrary, however, one report out of Japan last year claimed that Watson aided Tokyo researchers in detecting a type of leukemia, helping to save a patient's life. Specifically, according to a report in New Delhi Television Limited (NDTV), the disease had gone undetected using conventional methods. The 60-year-old female patient had been mystifying medical professionals from Japan after treatment—and all previous treatment—being prescribed for the condition was proving ineffective, according to a story in the International Business Times. But in just minutes, the Watson machine concluded that the patient suffered from a rare type of leukemia after her genetic information was compared with 20 million clinical oncology studies, which had been uploaded to its system by doctors from the University of Tokyo's Institute of Medical Science.

The patient was initially diagnosed with acute myeloid leukemia, which she was unsuccessfully treated for with several different anti-cancer drugs, leaving doctors confounded, a member of the medical team said in a separate report from last week. As such, the medical team decided to use the Watson system, which, on analyzing the data available, concluded the patient suffered from another form of leukemia and recommended a different treatment which was successful.

Along with MD Anderson, IBM Watson early on developed a partnership with the renowned cancer hospital Memorial Sloan Kettering. According to IBM officials, “Memorial Sloan Kettering clinicians and analysts are partnering with IBM to train Watson Oncology to interpret cancer patients’ clinical information and identify individualized, evidence-based treatment options that leverage our specialists’ decades of experience and research.”

But according to STAT, “Several doctors said Memorial Sloan Kettering’s training injects bias into the system, because the treatment recommendations it puts into Watson don’t always comport with the practices of doctors elsewhere in the world.”

Nevertheless, Watson continues to make partnerships and relationships around the world. It just announced this week in fact that it had reached a 10-year research partnership agreement with MIT worth $240 million. The two organizations will create a joint artificial intelligence lab with the aim to “advance AI hardware, software, and algorithms related to deep learning and other areas; increase AI’s impact on industries, such as healthcare and cybersecurity; and explore the economic and ethical implications of AI on society,” according to officials.

Get the latest information on Health IT and attend other valuable sessions at this two-day Summit providing healthcare leaders with educational content, insightful debate and dialogue on the future of healthcare and technology.

Learn More

Topics

News

Study will Leverage Connecticut HIE to Help Prevent Suicides

A new study will aim to leverage CTHealthLink, a physician-led health information exchange (HIE) in Connecticut, to help identify the factors leading to suicide and to ultimately help prevent those deaths.

Duke Health First to Achieve HIMSS Stage 7 Rating in Analytics

North Carolina-based Duke Health has become the first U.S. healthcare institution to be awarded the highest honor for analytic capabilities by HIMSS Analytics.

NIH Releases First Dataset from Adolescent Brain Development Study

The National Institutes of Health (NIH) announced the release of the first dataset from the Adolescent Brain Cognitive Development (ABCD) study, which will enable scientists to conduct research on the many factors that influence brain, cognitive, social, and emotional development.

Boston Children's Accelerates Data-Driven Approach to Clinical Research

In an effort to bring a more data-driven approach to clinical research, Boston Children’s Hospital has joined the TriNetX global health research network.

Paper Records, Films Most Common Type of Healthcare Data Breach, Study Finds

Despite the high level of hospital adoption of electronic health records and federal incentives to do so, paper and films were the most frequent location of breached data in hospitals, according to a recent study.

AHA Appoints Senior Advisor for Cybersecurity and Risk

The American Hospital Association (AHA) has announced that John Riggi has joined the association as senior advisor for cybersecurity and risk.