Skip to content Skip to navigation

Research: Physicians Outperform Computer Algorithms in Diagnostic Accuracy

October 11, 2016
by Heather Landi
| Reprints

While much has been made about the potential for diagnostic software to make accurate clinical diagnoses, in a head-to-head comparison with human doctors, researchers found that physicians made a correct diagnosis more than twice as often.

According to a research letter published in JAMA Internal Medicine, a research team, from Harvard Medical School, Brigham and Women’s Hospital and The Human Diagnosis Project, conducted a head-to-head comparison of physicians with symptom-checker apps and websites that help patients with self-diagnosis. The research builds on a previous evaluation of the diagnostic accuracy of 23 symptom checkers. For this study, the research team compared the diagnostic performance of physicians with symptom checkers for those same 45 vignettes using the digital platform Human Dx.

The research letter cites The Institute of Medicine recently highlighting that physician diagnostic error is common and information technology may be part of the solution. “Given the advancements in computer science, computers may be able to independently make accurate clinical diagnoses. While studies have compared computer versus physician performance for reading electrocardiograms, the diagnostic accuracy of computers versus physicians remains unknown. To fill this gap in knowledge, we compared the diagnostic accuracy of physicians with computer algorithms called symptom checkers,” the research authors wrote.

In the study, 234 internal medicine physicians were asked to evaluate 45 clinical cases, involving both common and uncommon conditions with varying degrees of severity. For each scenario, physicians had to identify the most likely diagnosis along with two additional possible diagnoses. Each clinical vignette was solved by at least 20 physicians. Of the 234 physicians who solved at least one vignette, 90 percent were trained in internal medicine and 52 percent were fellows or residents.

Given that physicians provided free text responses, two physicians hand-reviewed the submitted diagnoses and independently decided whether the participant listed the correct diagnosis first or in the top three diagnoses.

The researchers reported that physicians listed the correct diagnosis first 72 percent of the time, while the online tools listed the correct diagnosis just 34 percent of the time. Physicians outperformed the symptom-checker apps and websites by a margin of more than 2 to 1.

The physicians and the computer programs were able to include more than one ailment in their differential diagnosis. So, the researchers also compared how often the correct diagnosis was among the top three responses. Physicians made the correct diagnosis among their top three possibilities 84 percent of the time, while the digital symptom-checkers only did so 51 percent of the time, the researchers reported.

The difference between physician and computer performance was most dramatic in more severe and less common conditions. It was smaller for less acute and more common illnesses.

"While the computer programs were clearly inferior to physicians in terms of diagnostic accuracy, it will be critical to study future generations of computer programs that may be more accurate," senior investigator Ateev Mehrotra, an associate professor of health care policy at HMS, said.

Despite outperforming the machines, physicians still made errors in about 15 percent of cases. Researchers say developing computer-based algorithms to be used in conjunction with human decision-making may help further reduce diagnostic errors.

"Clinical diagnosis is currently as much art as it is science, but there is great promise for technology to help augment clinical diagnoses," Mehrotra said. "That is the true value proposition of these tools."

 

Get the latest information on Mobile Health and attend other valuable sessions at this two-day Summit providing healthcare leaders with educational content, insightful debate and dialogue on the future of healthcare and technology.

Learn More

Topics

News

Lenovo Health and Orbita Launch Voice-Enabled Home Health Assistant Technology

North Carolina-based health IT company Lenovo Health and Orbita, a Boston-based connected home healthcare company, launched a virtual home care solution and showcased the technology at HIMSS17 in Orlando.

Phase 2 Winners Chosen in ‘Move Health Data Forward’ Challenge

The Office of the National Coordinator for Health Information Technology has announced five winners in Phase 2 of the “Move Health Data Forward” Challenge, a contest to develop solutions to help with the flow of health information.

National Association for Trusted Exchange Unveils FHIR-Based Solution for Data Sharing

At the HIMSS17 conference in Orlando on Monday, The National Association for Trusted Exchange (NATE) unveiled NATE’s Blue Button Directory (NBBD) and is demonstrating it as part of the Federal Health Architecture’s demonstrations in the HIMSS17 Interoperability Showcase.

Health Catalyst Incorporates Regenstrief’s NLP Solution in Its Analytics Platform

At the HIMSS17 conference in Orlando, the nonprofit Regenstrief Institute announced a partnership with analytics vendor Health Catalyst involving Regenstrief's artificial intelligence-powered text analytics technology.

Survey: Cybersecurity Getting More Attention at the C-Suite and Board Level

Cybersecurity has been elevated to a central concern for healthcare providers, with more attention at the board level and the C-suite, according to a new survey by Orem, Utah-based KLAS Research and the College of Healthcare Information Management Executives (CHIME). The study found that 42 percent of organizations have a vice president or C-level official in charge of cybersecurity and for 39 percent of organizations, the head of cybersecurity is at the director level.

Partnership for Health IT Patient Safety Focuses on Patient Identification

The Partnership for Health IT Patient Safety has rolled out its second set of Safe Practice Recommendations with a focus on reducing patient misidentification.