Consumer Ratings Do Not Reflect Medical App Accuracy

A new study suggests that positive user reviews, comments, and ratings of popular medical apps do not necessarily mean that they actually work.

Researchers at the University of Vermont (UVM, Burlington, USA), Johns Hopkins University (JHU; Baltimore, MD, USA), and other institutions conducted a study of all user reviews of the now-defunct AuraLife Instant Blood Pressure (IBP) app in order to examine if they correctly reflected the proven inaccuracy of the unregulated blood pressure (BP) measuring app, to understand the drivers for high and low ratings, and to understand if company disclaimers prevented medical use.

To do so, 261 iTunes app reviews for IBP were downloaded and assessed for themes by two reviewers, with summary statistics for themes tabulated with their associated star ratings. The average star rating of the latest version of the app was four out of five stars, and 59% of the reviews assigned the app five stars. Commentary praising the accuracy of the app based on anecdotal experience comprised 42% of the reviews, and 10% of the reviews mention inaccuracy. Six reviews came from people who claimed to be health care professionals, who gave an average rating of 4.2 stars out of five.

While the IBP app included a disclaimer that it shouldn't be used as a medical device and is intended for recreational purposes only, 24 reviews (10%) claimed to use the app for medical purposes, with 11 people using it to manage their high BP treatments, one using it to manage kidney disease, and another using it to monitor BP after a heart transplant. Eleven reviews came from people who said their health care provider approved of the app. Two people said professionals disapproved of the app, and gave a one-star rating. The study was published on June 7, 2018, in the Journal of the American Medical Informatics Association.

“People tend to trust user reviews when shopping online and use them to decide which products to purchase, but that doesn't cut it for medical apps. There are certain thresholds of accuracy that need to be maintained, and a five-star rating doesn't replace clinical validation studies and FDA review,” said study author Timothy Plante, MD, MHS, of the University of Vermont. “Our take-home message to health care providers is to make sure you hold apps to the highest standards, and not just give out recommendations willy-nilly for apps that you think are neat.”

The IBP app sold for USD 4.99 on iTunes and was downloaded 140,000 times before it was withdrawn. It ranked among the best-selling apps on all of iTunes for much of its stretch on the app store. After the app was withdrawn and following the study, the U.S. Federal Trade Commission settled with AuraLife for USD 600,000, with payment suspended for lack of funds in the company. Many other medical apps are still available for consumers, some downloading as many as 1,000,000 times each.

Related Links:
University of Vermont
Johns Hopkins University