Advances in computing have led to a flood of medical devices powered by artificial intelligence and machine learning (AI/ML) that allow doctors, nurses, and even patients themselves to better identify and track a wide variety of health issues.

There’s a program that analyzes video of a child to help determine if they have autism. There’s a stethoscope that uses software to amplify the faint sound of heart murmurs. Medical professionals are using AI/ML-enabled devices that locate polyps in a patient’s colon, analyze whether their coronary arteries have narrowed, and visualize the best spot to begin brain surgery. And patients are using smartphone apps to track their kidneys, irregular heartbeats, and sleep patterns.

“The responsibility for ensuring medical devices are safe
and effective falls on officials at the FDA.”

With more than 500 AI/ML-enabled medical devices already on the market and hundreds more in development, the responsibility for ensuring they are safe and effective falls on officials at the U.S. Food and Drug Administration (FDA). This rapid increase in the use of AI/ML-enabled medical devices raises new questions regarding the evaluation of the performance of these novel technologies in real-world settings. Many of these questions are difficult to answer because of a lack of access to clinical, real-world data suitable for studying the performance of the devices and the lack of proper methodologies to analyze and distill the data into actionable metrics.

To help address some of these key questions, the FDA will collaborate with Dr. Yelena Yesha, head of the AI/ML Unit at the University of Miami’s Frost Institute for Data Science & Computing, who has a Ph.D. in computer science and has collaborated with scientists at the FDA before. Yesha signed a 5-year research agreement with the Office of Science and Engineering Labs (OSEL) at the FDA’s Center for Devices and Radiological Health (CDRH) to build a computer-based system that can help evaluate the safety and efficacy of AI/ML-medical devices and the data they capture. Yesha is now assembling a team of collaborators including AI/ML experts at IDSC, doctors at the University of Miami Miller School of Medicine, and FDA scientists who already have research appointments at UM.

“We need to corroborate the AI findings
with underlying ‘ground-truth’ clinical data”

“Hundreds of AI-enabled devices and algorithms have been authorized in the past few years, and many in the field of radiology and image analysis. However, we need common pathways and methodology to evaluate their impact, safety, and reproducibility of results, as well as to corroborate the AI findings with underlying ‘ground-truth’ clinical data,” said Alexander McKinney, an M.D. and the Robert Shapiro Chair of Radiology at the Miller School who will be part of the research team. “This landmark research collaboration agreement between the Office of Science and Engineering Labs (OSEL) at the FDA’s Center for Devices and Radiological Health (CDRH) and the University of Miami will lead to a common mechanism to share, curate, and study AI/ML algorithms’ performance on images and imaging data, as well as study its relevance to care and outcomes.”

The UM team will focus first on developing a “testbed”—a computational platform that is cut off from any active medical devices but mimics the way they operate and the data they produce. That will allow the UM team to finetune their approach before testing their analysis program on real-world devices. The testbed will also evaluate their programs using real-world datasets to gauge the efficacy of the devices.

Streamlining and automating the analysis of new medical devices will become even more critical in the years to come. The FDA already regulates 20% of products purchased by U.S. consumers, and the speed of technology means that more products will be powered by AI/ML in years to come. The research collaboration will benefit the FDA, not only for the specific applications that are targeted in data collection, but also as archetypes to answer more fundamental questions about how the safety, relevance, reliability, and effectiveness of AI/ML-enabled devices can be optimized.

“One of the goals is to identify ‘technological gaps’
in the devices that are being tested.”

It’s not just the government that will benefit from the testing program developed by the UM team. One of the goals of the research collaboration is to develop datasets, analytics, and a testbed that can also identify “technological gaps” in the devices that are being tested. Yesha said that will help manufacturers improve their products. The datasets and analytics that will be created by the new testing program will also provide, for the first time, a comprehensive view of patient health in the short- and long-term. As more data is accumulated, researchers will have a more precise picture of what each AI/ML-enabled device is doing for patients.

Yesha stresses that’s their ultimate goal: to improve the quality of medical care delivered to patients. Yesha has studied a wide variety of topics throughout her career including cybersecurity, satellite imagery, e-commerce, and climate change. But she’s always remained active in medical research, including a decade of training computers to identify lung cancers from X-Rays. Yesha said those projects are the most fulfilling because of the practical effects they have on the health of humanity.

“In everything we do in medicine, my goal is to take a patient-centric approach. The patient is the center of the universe,” she said. “That’s the goal of any research our group is doing.”