Survey: Variety, Not Volume, the Biggest Challenge to Analyzing Big Data | Healthcare Informatics Magazine | Health IT | Information Technology Skip to content Skip to navigation

Survey: Variety, Not Volume, the Biggest Challenge to Analyzing Big Data

July 9, 2014
by Rajiv Leventhal
| Reprints

While the huge volume of big data in healthcare continues to generate headlines, a new report finds that it’s the diverse types of data that’s a bigger challenge to data scientists and is causing them to “leave data on the table.”

According to a new survey by the Waltham, Mass.-based computational database company Paradigm4, nearly three-quarters of data scientists—71 percent—said big data had made their analytics more difficult, and data variety, not volume of data from electronic health records (EHRs), connected mobile devices, genomic sequencers and sensors, was to blame.

The survey also showed that 36 percent of data scientists say it takes too long to get insights because the data is too big to move to their analytics software. These issues cause data scientists to omit data from analyses and prevent them from maximizing the value of their work.

Additional results of the survey, which were generated from 111 data scientists in the U.S., found that 91 percent of data scientists said they're using complex analytics on their big data now or plan to within the next two years. Forty-nine percent of respondents said they're finding it more difficult to fit their data into relational database tables. And 40 percent said their biggest problem they face in gaining insights from their big data is managing new types and sources of data.

 “The increasing variety of data sources is forcing data scientists into shortcuts that leave data and money on the table,” Marilyn Matz, CEO of Paradigm4, said in a statement. “The focus on the volume of data hides the real challenge of data analytics today. Only by addressing the challenge of utilizing diverse types of data will we be able to unlock the enormous potential of analytics.”

Get the latest information on Health IT and attend other valuable sessions at this two-day Summit providing healthcare leaders with educational content, insightful debate and dialogue on the future of healthcare and technology.

Learn More



KLAS Research: Small Hospitals’ Buying Decisions Impacting EMR Market Share

A new KLAS Research report tracks shifts in electronic medical record (EMR) vendor market share among acute care hospitals, and finds that smaller hospitals are seeking technology solutions that meet their needs and limited budgets, and these contracts are making a mark on the EMR market.

Survey: Majority of Providers Predict Success for New Generic Drug Company, Project Rx

Back in January, four health systems, in consultation with the VA, announced a collaboration to develop a new, not-for-profit generic drug company. A survey has found that 90 percent of providers say they would become customers of the new venture.

Personalized Medicine Awareness Low Among U.S. Adults, Survey Finds

Genetics and personalized medicine are not top of mind for the general public in the U.S., according to a recent survey from GenomeWeb and the Personalized Medicine Coalition.

Industry Organizations Praise Senate Passage of VA Mission Act

The U.S. Senate on Wednesday passed, by a vote of 92-5, a major Veterans Affairs (VA) reform bill that includes health IT-related provisions to improve health data exchange between VA healthcare providers and community care providers.

NIH Issues Funding Announcement for All of Us Genomic Research Program

The National Institutes of Health’s (NIH) “All of Us” Research Program has issued a funding announcement for genome centers to generate genotype and whole genome sequence data from participants’ biosamples.

MGMA: Physician Compensation Data Illustrates Nationwide PCP Shortage

Primary care physicians’ compensation rose by more than 10 percent over the past five years, representing an increase which is nearly double that of specialty physicians’ compensation over the same period, according to the Medical Group Management Association (MGMA).