Highlighting the need for a more nuanced understanding of app quality in the digital health landscape, the study titled Don’t Judge a Book or Health App by its Cover was led by researchers at Ulster University’s School of Computing and delivered in partnership with the Organisation for the Review of Care and Health Apps (ORCHA).
With over 350,000 healthcare apps available to download at the click of a button covering topics from diabetes to sleep to mental health, the study analysed data from 1,500 health apps across various platforms, assessing their quality against user ratings and download counts.
In an era where digital health apps are increasingly relied upon for managing various aspects of health and wellness, and with consumers potentially swayed by popular perception, the research team explored the disconnect between app popularity metrics and safe, high-quality digital health apps and the resulting significant implications for healthcare providers, policymakers, and app developers.
Professor Raymond Bond, Professor of Human Computer Systems at Ulster University and Chair of the PhD research project, said:
"Our research underscores the importance of looking beyond metrics like user ratings and the number of app downloads when evaluating digital health apps. While these metrics may influence user perceptions, our research shows that they do not necessarily reflect the quality, effectiveness, safety, or reliability of these healthcare apps.
"We need to shift the focus from popularity to quality. Consumers deserve access to high-quality digital health interventions that are evidence-based, user-friendly, and designed to enhance health and wellbeing outcomes. Relying solely on user ratings and download counts could lead to misguided decisions and, eventually, undermine public trust in digital health technologies."
By recognising the limitations of traditional metrics and prioritising rigorous evaluation processes, stakeholders, including healthcare professionals who recommend apps to patients, can ensure the applications they are recommending meet the highest standards of quality and contribute meaningfully to patient care and well-being.
Professor Maurice Mulvenna, Professor of Computer Science at Ulster University, added:
“As the digital health landscape continues to evolve, studies like these serve as a vital reminder that appearances can be deceptive. In delving deeper into the substance of digital health apps and by prioritising evidence-based evaluation, we can help consumers to make informed decisions and harness the full potential of technology in advancing health outcomes. We are pleased to see our Ulster University-led research reach a broad audience that plays a role in safeguarding customers and patients as they make use of the growing digital health app marketplace.”
Conducted as part of a PhD programme in 2021-2024, the research examined a unique dataset of over 1500 digital health apps provided by ORCHA, assessing the level of evidence behind the app, whether it is user-friendly and accessible, and areas including data protection and privacy.
The study, which was published in the PLOS ONE journal (https://doi.org/10.1371/journal.pone.0298977), was jointly funded by ORCHA and the Department for the Economy as part of the Co-operative Awards in Science and Technology (CAST) studentship.
Liz Ashall-Payne, Founding CEO of ORCHA said:
“Our collaboration with Ulster University has shed light on a crucial issue in the digital health landscape. User ratings and download numbers are not reliable indicators of an app’s quality. This research emphasises the need for comprehensive evaluation to ensure that digital health apps meet the highest standards of safety, effectiveness, and reliability. By prioritising rigorous assessments, we can guide healthcare providers and consumers toward better choices, ultimately improving health outcomes and building trust in digital health solutions.”
Maciej Hyzy, PhD Researcher added:
"Taking part in Ulster University’s PhD programme and collaborating with ORCHA gave me an opportunity to be part of important research in the area of digital health. As a result of this collaboration and research we increased our understanding of digital health apps by analysing their assessment data. Research like this shows us that we need to rethink metrics by which we judge digital health apps."