Any organization deploying algorithms in high-impact areas must track risks of harm to stakeholders. The statement comes from the CEO of a risk consulting and algorithmic auditing firm and fellow authors of a paper on algorithmic risk auditing around the world. MIT Sloan Management Review.
It’s a compelling concept, that of auditing algorithmic risks; However, if we were truly prepared to audit such a risk, it might not have been evident at a November 20 meeting of the U.S. Food and Drug Administration’s Digital Health Advisory Committee (DHAC ), where stakeholders spent time in a Public meeting broadcast on YouTube discuss the regulation of generative AI and the use of restricted AI tools.
“I’ve looked everywhere and I don’t believe there is a single health system in the United States that can validate an AI algorithm implemented in a clinical care system,” FDA commissioner said , Robert Califf, MD, at the FDA opening. DHAC meeting. Califf is no stranger to medical technology.
Prior to working at the FDA, Califf was head of medical strategy and senior advisor at Alphabet Inc. He contributed to strategy and policy for Alphabet’s healthcare subsidiaries, Verily Life Sciences and Google Health. With the FDA currently undergoing restructuring, Califf and others are monitoring what changes the Trump administration may make.
AI everywhere
In the drug and biologics space, almost every application incorporates AI at some point in the development process, Califf said at the DHAC meeting. Perhaps in contrast, AI used routinely in clinical care presents a range of complexities and variables that experts at the American College of Radiology (ACR), for example, are currently trying to resolve.
Bernardo Bizzo, MD, PhD, diagnostic radiologist and senior director of AI activities at Mass General Brigham, was among those who provided their perspective on the field’s work on AI quality and safety . Bizzo is associate scientific director at the ACR Data Science Institute.
For DHAC, Bizzo highlighted ACR’s national AI quality assurance program, ARCH-AI, which stands for ACR Recognized Center for Healthcare-AI. Beneath ARCH-AI is Assess-AI, Bizzo explained.
Evaluate-AI is a registry through the ACR that enables real-world monitoring of imaging-based AI models deployed in a clinical workflow. It allows local performance measures to be compared to national benchmarks and facilities with similar characteristics (region, facility type, trauma level, urban/rural, Bizzo explained.
Bizzo added that the program is currently being tested at 15 sites with four AI vendors and platforms engaged for two FDA-approved clinical use cases, intracranial hemorrhage and pulmonary embolism on CT cases.
Narrow AI tools characterize the current state, Bizzo said, but the future state is generative AI. Audience members did, however, have questions about determining clinically safe thresholds for use cases and determining specific elements and metrics to monitor for safety and effectiveness. Bizzo said those discussions are ongoing.
Episode 10
With all of this as food for thought, we bring you this episode of Keeping Up With the Radiologists. Tessa Cook, MD, PhD, director of Penn Medicine’s Radiology Practice Transformation Center who develops and drives innovation initiatives, and Saurabh (Harry) Jha, MD, are joined by Hugh Harvey, MD, managing director of Hardian Health Consultants.
Harvey is among those working to commercialize software as a medical device (SaMD) and SaMD systems. The latest concerns Guildford Street Laboratories, a spin-out from University College London (UCL), and its blood test for Parkinson’s disease, PD Predict, which Hardian announced received FDA Breakthrough Device designation in November. The diagnostic test measures multiple biomarkers using machine learning analysis.
Regulatory spectrum
“The question of all regulation is whether we are over-regulating, whether we are under-regulating, what is the appropriate level of regulation, and what is the purpose of regulating medical devices and artificial intelligence,” Jha asks at the beginning of the episode. Inside, jokes ensue about Europe versus the United States and who could do better with their people, but that’s up to the listeners to decide.
“Regulation is a spectrum from total over-regulation, meaning banning certain things, to zero regulation to let chaos run rampant,” Harvey said. “And I think anyone who works in the medical sector would agree that some regulation is needed for things that will impact patients, hospital care and general practice. If we can all agree on that premise of basis that there must be checks and balances — it can’t be a total free for all — we can’t outright ban new technologies because we’re afraid of them.”
Where do we want to put the needle on the regulation pressure gauge? Listen to the panel discuss: CE marking versus FDA 510(k) and areas where obtaining regulatory clearance is more difficult, according to Harvey. Hear the fundamental reason why there has yet to be a single regulatory approval for a Class 3 autonomous AI system. Hear where the “cutting edge” stands today in relation to algorithm development ten years ago.
More impressions
{06:21:23} European legislation on AI
{06:45:23} Decision to pilot an AI algorithm
{08:57:10} Proof and rigor
{10:34:12} Ask for standards
{14:34:21} Transparency
{17:45:25} Algorithms are the new drugs
{20:20:22} Risks that arise through software
{22:14:19} Trust
{25:37:14} AI inputs continue to change
{32:53:20} Hospital Quality Management Systems
{35:49:19} Managing AI in the real world
{37:28:03} Study design AI vs radiologists
{41:23:15} Lack of great tests
{50:20:19} AI Failure Promise
{54:05:01} Efficiency gains
Guests:
Hugh Harvey, MD, is Managing Director of Hardian Health consultants, a UK-based clinical digital consultancy focused on leveraging technology in healthcare markets through clinical strategy, scientific validation, regulation, health economics and intellectual property.
Hosts:
Saurabh (Harry) Jha, MD, is an associate professor of radiology at the Hospital of the University of Pennsylvania. Jha earned a master’s degree in health policy research from the Leonard Davis Institute at the University of Pennsylvania. He received his medical degree from the United Medical and Dental Schools of Guy, King and St. Thomas Hospitals. Jha developed Value of Imaging, a set of radiology educational resources.
Tessa Cook, MD, PhD, is a cardiovascular radiologist and board-certified clinical informaticist at Penn Medicine. Cook is director of Penn Medicine’s 3D and Advanced Imaging Laboratory and director of Penn Medicine’s Center for Practice Transformation in Radiology, which develops and drives innovation initiatives to improve radiology care delivery. This includes AI.
This episode of Keeping Up With the Radiologists is brought to you by TanteMinnie.com in collaboration with Penn Radiology. The series is also available on Spotify, YouTube MusicAnd Apple Podcasts. Come back for new episodes!