IBM Watson’s Shortcomings Investigated in STAT Report | Healthcare Informatics Magazine | Health IT | Information Technology Skip to content Skip to navigation

IBM Watson’s Shortcomings Investigated in STAT Report

September 7, 2017
by Rajiv Leventhal
| Reprints

An investigation into IBM Watson by STAT News has revealed that the artificial intelligence supercomputer has not lived up to its potential.

The piece, penned by Casey Ross and Ike Swetlitz, examined Watson for Oncology’s use, marketing, and performance in hospitals across the world, from South Korea to Slovakia to South Florida. The in-depth article included interviews with doctors that have deployed Watson at prominent healthcare institutions as well as other healthcare experts—many of whom seem far from thrilled about the results Watson has delivered so far.  

The idea behind Watson for Oncology is to help physicians quickly identify key information in a patient’s medical record, surface relevant articles and explore treatment options to reduce unwanted variation of care and give time back to their patients, according to the website of Watson Health, a unit of IBM that was launched at the 2015 HIMSS conference and which employs more than 7,000 people.

But according to the STAT article, Watson for Oncology’s “treatment recommendations are not based on its own insights from these data. Instead, they are based exclusively on training by human overseers, who laboriously feed Watson information about how patients with specific characteristics should be treated.” In fact, investigators said that while IBM sales executives say that Watson for Oncology possesses the ability to identify new approaches to cancer center, in reality, “the system doesn’t create new knowledge and is artificially intelligent only in the most rudimentary sense of the term.”

Despite being supported with major funding and incredible brand marketing, Watson is falling short in its ability to transform cancer care, sources in the article said. According to the STAT piece, “Three years after IBM began selling Watson to recommend the best cancer treatments to doctors around the world, a STAT investigation has found that the supercomputer isn’t living up to the lofty expectations IBM created for it. It is still struggling with the basic step of learning about different forms of cancer. Only a few dozen hospitals have adopted the system, which is a long way from IBM’s goal of establishing dominance in a multibillion-dollar market. And at foreign hospitals, physicians complained its advice is biased toward American patients and methods of care.”

What’s more the authors attested, “The actual capabilities of Watson for Oncology are not well-understood by the public, and even by some of the hospitals that use it. It’s taken nearly six years of painstaking work by data engineers and doctors to train Watson in just seven types of cancer, and keep the system updated with the latest knowledge.”

Watson, the Jeopardy!-playing supercomputer, is built to mirror the human learning process through the power of cognition. Its executives, some who were interviewed for the STAT piece, attest that Watson is till on an upward trajectory and is a major part of IBM’s business. They admitted that Watson is still in its infancy but that it’s growing rapidly; by the end of this year, noted the executives, “The system will offer guidance about treatment for 12 cancers that account for 80 percent of the world’s cases.”

Watson has had both successes and failures reported in the media in recent months. A Forbes report released the day before IBM’s global CEO Ginni Rometty's keynote at HIMSS17 revealed that the partnership between IBM and MD Anderson Cancer Center, part of the University of Texas, soured to the point where the $62 million project for the cancer center to deploy Watson had been scratched.

Bizarrely, the Forbes report also noted that MD Anderson is still on the hook to pay for the whole thing and that its leader in charge of the project never got approval for the implementation from the center's IT department. Questions remain regarding MD Anderson's current relationship with IBM,

Lynda Chin, M.D., who oversaw the Watson project at MD Anderson before it fell apart, told STAT reporters that it was quite challenging to make the technology functional in healthcare. “Teaching a machine to read a record is a lot harder than anyone thought,” she told STAT, noting how her team spent countless hours trying to get the machine to deal with the idiosyncrasies of medical records.

To the contrary, however, one report out of Japan last year claimed that Watson aided Tokyo researchers in detecting a type of leukemia, helping to save a patient's life. Specifically, according to a report in New Delhi Television Limited (NDTV), the disease had gone undetected using conventional methods. The 60-year-old female patient had been mystifying medical professionals from Japan after treatment—and all previous treatment—being prescribed for the condition was proving ineffective, according to a story in the International Business Times. But in just minutes, the Watson machine concluded that the patient suffered from a rare type of leukemia after her genetic information was compared with 20 million clinical oncology studies, which had been uploaded to its system by doctors from the University of Tokyo's Institute of Medical Science.

The patient was initially diagnosed with acute myeloid leukemia, which she was unsuccessfully treated for with several different anti-cancer drugs, leaving doctors confounded, a member of the medical team said in a separate report from last week. As such, the medical team decided to use the Watson system, which, on analyzing the data available, concluded the patient suffered from another form of leukemia and recommended a different treatment which was successful.

Along with MD Anderson, IBM Watson early on developed a partnership with the renowned cancer hospital Memorial Sloan Kettering. According to IBM officials, “Memorial Sloan Kettering clinicians and analysts are partnering with IBM to train Watson Oncology to interpret cancer patients’ clinical information and identify individualized, evidence-based treatment options that leverage our specialists’ decades of experience and research.”

But according to STAT, “Several doctors said Memorial Sloan Kettering’s training injects bias into the system, because the treatment recommendations it puts into Watson don’t always comport with the practices of doctors elsewhere in the world.”

Nevertheless, Watson continues to make partnerships and relationships around the world. It just announced this week in fact that it had reached a 10-year research partnership agreement with MIT worth $240 million. The two organizations will create a joint artificial intelligence lab with the aim to “advance AI hardware, software, and algorithms related to deep learning and other areas; increase AI’s impact on industries, such as healthcare and cybersecurity; and explore the economic and ethical implications of AI on society,” according to officials.

The Health IT Summits gather 250+ healthcare leaders in cities across the U.S. to present important new insights, collaborate on ideas, and to have a little fun - Find a Summit Near You!


/news-item/analytics/ibm-watson-s-shortcomings-investigated-stat-report
/blogs/david-raths/analytics/amia-charts-course-learning-health-system

AMIA Charts Course to Learning Health System

| Reprints
Initiative seeks to create virtuous cycle where clinical practice is not distinct from research

In September 2015, at AcademyHealth’s Concordium 2015 meeting in Washington, D.C., I saw a great presentation by Peter Embi, M.D., who was then an associate professor and vice chair of biomedical informatics as well as associate dean for research informatics and the chief research information officer at the Wexner Medical Center at Ohio State University. 

That day Dr. Embi outlined some of the limitations of the traditional approach to evidence-based medicine —  that it is a research/practice paradigm where the information flow is unidirectional, and clinical practice and research are distinct activities, with the research design as an afterthought. “We want to leverage information at the point of care and in engagements with patients so we can systematically learn. That is what the learning health system is all about,” Embi said.

But in the current model, he noted, there is little consideration of research during planning of health systems. That limits the ability to invest in and leverage clinical resources to advance research. Also, there are no financial incentives for non-researchers to engage in research. Research as an afterthought also leads to regulatory problems and wasted investments.

Embi argued for moving from “evidence-based medicine” to an “evidence-generating medicine” approach, which he defined as the systematic incorporation of research and quality improvement into the organization. Rather than findings flowing only from research done looking back at historical data, this approach creates a virtuous cycle where clinical practice is not distinct from research.

Flash forward to 2019 and Dr. Embi is now president & CEO of Regenstrief Institute Inc., vice president for learning health systems at IU Health, and chairman of the Board of Directors of the American Medical Informatics Association (AMIA). And he is still advocating for a shift to evidence-generating medicine. He and AMIA colleagues recently published a paper in JAMIA offering more than a dozen recommendations for public policy to facilitate the generation of evidence across physician offices and hospitals now that the adoption of EHRs is widespread.

The paper cites several examples of current high-visibility research initiatives that depend on the EGM approach: the All of Us Research Program and Cancer Moonshot initiative, the Health Care Systems Research Collaboratory, and the development of a national system of real-world evidence generation system as pursued by such groups as the US Food & Drug Administration (FDA), Patient-Centered Outcomes Research Institute (PCORI), National Institutes of Health (NIH), and other federal agencies.

The paper makes several recommendations for policy changes, including that the Trump administration should faithfully implement 2018 Revisions to the Common Rule as well as establish the 21st Century Cures-mandated Research Policy Board. The administration must implement this provision to better calibrate and harmonize our sprawling and incoherent federal research regulations.

Another recommendation is that the HHS Office of Civil Rights (OCR) should refine the definition of a HIPAA Designated Record Set (DRS) and ONC should explore ways to allow patients to have a full digital export of their structured and unstructured data within a Covered Entity’s DRS in order to share their data for research. In addtion, regulators should work with stakeholders to develop granular data specifications, including metadata, and standards to support research for use in the federal health IT certification program.

The AMIA authors also suggest that CMS leverage its Quality Payment Program to reward clinical practice Improvement Activities that involve research components. This would encourage office-based physicians to invest time and resources needed to realize EGM, they say.

Based on the paper’s findings, AMIA is launching a new initiative focused on advancing informatics-enabled improvements for the U.S. healthcare system. The organization says that a multidisciplinary group of AMIA members will develop a national informatics strategy, policy recommendations, and research agenda to improve:

• how evidence is generated through clinical practice;

• how that evidence is delivered back into the care continuum; and

• how our national workforce and organizational structures are best positioned to facilitate informatics-driven transformation in care delivery, clinical research, and population health.

A report detailing this strategy will be unveiled at a December 2019 conference in Washington, D.C.

 

 

More From Healthcare Informatics

/news-item/analytics/definitive-healthcare-acquires-himss-analytics-data-services

Definitive Healthcare Acquires HIMSS Analytics’ Data Services

January 16, 2019
by Rajiv Leventhal, Managing Editor
| Reprints

Definitive Healthcare, a data analytics and business intelligence company, has acquired the data services business and assets of HIMSS Analytics, the organizations announced today.

The purchase includes the Logic, Predict, Analyze and custom research products from HIMSS Analytics, which is commonly known as the data and research arm of the Healthcare Information and Management Systems Society.

According to Definitive officials, the acquisition builds on the company’s “articulated growth strategy to deliver the most reliable and consistent view of healthcare data and analytics available in the market.”

Definitive Healthcare will immediately begin integrating the datasets and platform functionality into a single source of truth, their executives attest. The new offering will aim to include improved coverage of IT purchasing intelligence with access to years of proposals and executed contracts, enabling transparency and efficiency in the development of commercial strategies.

Broadly, Definitive Healthcare is a provider of data and intelligence on hospitals, physicians, and other healthcare providers. Its product suite its product suite provides comprehensive data on 8,800 hospitals, 150,000 physician groups, 1 million physicians, 10,000 ambulatory surgery centers, 14,000 imaging centers, 86,000 long-term care facilities, and 1,400 ACOs and HIEs, according to officials.

Together, Definitive Healthcare and HIMSS Analytics have more than 20 years of experience in data collection through exclusive methodologies.

“HIMSS Analytics has developed an extraordinarily powerful dataset including technology install data and purchasing contracts among other leading intelligence that, when combined with Definitive Healthcare’s proprietary healthcare provider data, will create a truly best-in-class solution for our client base,” Jason Krantz, founder and CEO of Definitive Healthcare, said in a statement.

Related Insights For: Analytics

/news-item/analytics/machine-learning-survey-many-organizations-several-years-away-adoption-citing

Machine Learning Survey: Many Organizations Several Years Away from Adoption, Citing Cost

January 10, 2019
by Heather Landi, Associate Editor
| Reprints

Radiologists and imaging leaders see an important role for machine learning in radiology going forward, however, most organizations are still two to three years away from adopting the technology, and a sizeable minority have no plans to adopt machine learning, according to a recent survey.

A recent study* by Reaction Data sought to examine the hype around artificial intelligence and machine learning, specifically in the area of radiology and imaging, to uncover where AI might be more useful and applicable and in what areas medical imaging professionals are looking to utilize machine learning.

Reaction Data, a market research firm, got feedback from imaging professionals, including directors of radiology, radiologists, chiefs of radiology, imaging techs, PACS administrators and managers of radiology, from 152 healthcare organizations to gauge the industry on machine learning. About 60 percent of respondents were from academic medical centers or community hospitals, while 15 percent were from integrated delivery networks and 12 percent were from imaging centers. The remaining respondents worked at critical access hospitals, specialty clinics, cancer hospitals or children’s hospitals.

Among the survey respondents, there was significant variation in the number of annual radiology studies performed—17 percent performed 100-250 thousand studies each year; 16 percent performed 1 to 2 million studies; 15 percent performed 5 to 25 thousand studies; 13 percent performed 250 to 500 thousand; 10 percent performed more than 2 million studies a year.

More than three quarters of imaging and radiology leaders (77 percent) view machine learning as being important in medical imaging, up from 65 percent in a 2017 survey. Only 11 percent view the technology as not important. However, only 59 percent say they understand machine learning, although that percentage is up from 52 percent in 2017. Twenty percent say they don’t understand the technology, and 20 percent have a partial understanding.

Looking at adoption, only 22 percent of respondents say they are currently using machine learning—either just adopted it or have been using it for some time. Eleven percent say they plan to adopt the technology in the next year.

Half of respondents (51 percent) say their organizations are one to two years away (28 percent) or even more than three years away (23 percent) from adoption. Sixteen percent say their organizations will most likely never utilize machine learning.

Reaction Data collected commentary from survey respondents as part of the survey and some respondents indicated that funding was an issue with regard to the lack of plans to adopt the technology. When asked why they don’t ever plan to utilize machine learning, one respondent, a chief of cardiology, said, “Our institution is a late adopter.” Another respondent, an imaging tech, responded: “No talk of machine learning in my facility. To be honest, I had to Google the definition a moment ago.”

Survey responses also indicated that imaging leaders want machine learning tools to be integrated into PACS (picture archiving and communication systems) software, and that cost is an issue.

“We'd like it to be integrated into PACS software so it's free, but we understand there is a cost for everything. We wouldn't want to pay more than $1 per study,” one PACS Administrator responded, according to the survey.

A radiologist who responded to the survey said, “The market has not matured yet since we are in the research phase of development and cost is unknown. I expect the initial cost to be on the high side.”

According to the survey, when asked how much they would be willing to pay for machine learning, one imaging director responded: “As little as possible...but I'm on the hospital administration side. Most radiologists are contracted and want us to buy all the toys. They take about 60 percent of the patient revenue and invest nothing into the hospital/ambulatory systems side.”

And, one director of radiology responded: “Included in PACS contract would be best... very hard to get money for this.”

The survey also indicates that, among organizations that are using machine learning in imaging, there is a shift in how organizations are applying machine learning in imaging. In the 2017 survey, the most common application for machine learning was breast imaging, cited by 36 percent of respondents, and only 12 percent cited lung imaging.

In the 2018 survey, only 22 percent of respondents said they were using machine learning for breast imaging, while there was an increase in other applications. The next most-used application cited by respondents who have adopted and use machine learning was lung imaging (22 percent), cardiovascular imaging (13 percent), chest X-rays (11 percent), bone imaging (7 percent), liver imaging (7 percent), neural imaging (5 percent) and pulmonary imaging (4 percent).

When asked what kind of scans they plan to apply machine learning to once the technology is adopted, one radiologist cited quality control for radiography, CT (computed tomography) and MR (magnetic resonance) imaging.

The survey also examines the vendors being used, among respondents who have adopted machine learning, and the survey findings indicate some differences compared to the 2017 survey results. No one vendor dominates this space, as 19 percent use GE Healthcare and about 16 percent use Hologic, which is down compared to 25 percent of respondents who cited Hologic as their vendor in last year’s survey.

Looking at other vendors being used, 14 percent use Philips, 7 percent use Arterys, 3 percent use Nvidia and Zebra Medical Vision and iCAD were both cited by 5 percent of medical imaging professionals. The percentage of imaging leaders citing Google as their machine learning vendor dropped from 13 percent in 2017 to 3 percent in this latest survey. Interestingly, the number of respondents reporting the use of homegrown machine learning solutions increased to 14 percent from 9 percent in 2017.

 

*Findings were compiled from Reaction Data’s Research Cloud. For additional information, please contact Erik Westerlind at ewesterlind@reactiondata.com.

 

See more on Analytics

agario agario---betebet sohbet hattı betebet bahis siteleringsbahis