A computer-assisted needle misses its target, puncturing the spine. A diabetic patient goes rapidly downhill after a computer recommends an incorrect insulin dosage. An ultrasound fails to diagnose an obvious heart condition that is ultimately fatal.
These are just a few examples of incidents reported to the United States’ Food and Drug Administration involving health technology assisted by artificial intelligence (AI), and Australian researchers say they are an “early warning sign” of what could happen if regulators, hospitals and patients don’t take safety seriously in the rapidly evolving field.
“This is essentially showing us that when we’re putting in AI systems, we just need to be taking the safety of these systems really seriously,” Professor Farah Magrabi said.
Her team at the Australian Institute of Health Innovation at Macquarie University this month published a review of 266 safety events involving AI-assisted technology reported to the US watchdog. The article appeared in the Journal of the American Medical Informatics Association.
Only 16 per cent actually led to patients being harmed, but two-thirds were found to have the potential to cause harm, and 4 per cent were categorised as “near-miss events” in which users intervened.
Co-author Dr David Lyell said issues arose most commonly when users failed to enter the correct data, leading to an incorrect result, or misunderstood what the AI was actually telling them when it produced a result.
For example, one patient suffering a heart attack delayed medical care because an over-the-counter electrocardiogram device – which is not capable of detecting a heart attack – told them they had “normal sinus rhythm”.
“AI isn’t the answer; it’s part of a system that needs to support the provision of healthcare. And we do need to make sure that we have the systems in place that supports its effective use to promote healthcare for people,” Lyell said.
The researchers chose to analyse cases in the US, where the implementation of AI-enabled health devices is more advanced than in Australia. The US regulator has, to date, approved 521 artificial intelligence and machine learning-enabled medical devices, with 178 of those added in 2022.
In Australia, the Therapeutic Goods Administration (TGA) does not collect data about the number of approved devices in Australia that have AI or machine-learning components, but Magrabi said the regulator was taking the issue “very, very seriously”.
“AI isn’t the answer, it’s part of a system that needs to support the provision of health care.”
David Lyell, Australian Institute of Health Innovation, Macquarie University
AI has been used in healthcare devices for decades, but an explosion in data collection, computing power and advanced algorithms has opened up new frontiers, said David Hansen, chief executive of the CSIRO’s Australian E-Health Research Centre.
Sydney-based medical device start-up EMVision is one Australian company taking advantage of these advancements to develop a portable device for diagnosing stroke without the need for an MRI.
In the development stage, the company is using an advanced algorithm and high-powered computers to simulate stroke in numerous places in the brain, building up a database of synthetic images similar to MRIs which are then compared with real-life MRI and CT results from hospital clinical trials at Royal Melbourne, Liverpool and Princess Alexandra hospitals.
“We wouldn’t be able to do what we’re doing today if we didn’t have the [high-powered computer] infrastructure for the simulation,” said head of product development Forough Khandan.
Co-founder Scott Kirkland said the intention was not to completely replace CT and MRI scans, but to diagnose stroke in the first hour when treatment is most effective. The bedside device, set to be launched in 2025, will use a “traffic light” system based on a probability algorithm to help determine what type of stroke might have occurred.
“It’s better for an algorithm to give an ‘I don’t know’ than an incorrect answer, and have the wrong treatment and or triage process followed,” Kirkland said.
Radiology is at the forefront of the rapid adoption of AI in healthcare, especially in breast cancer screening and analysis of chest X-rays.
“A couple of years ago, almost no radiologist would say they use it, now a fair percentage would say that they use it in their daily work,” said clinical radiologist and AI safety researcher Dr Lauren Oakden Rayner.
Rayner, a member of the college, said the technology had many potential benefits, but Australian regulators and clinicians needed to better understand the risks of fully autonomous systems before putting them into hospitals, clinics and homes.
“Humans are legally and morally responsible for decision-making, and it’s taking some of that out of human hands,” she said. “There’s no reason autonomous AI systems can’t exist … but they obviously have to be tested very, very tightly.”
News
Scientists study lipids cell by cell, making new cancer research possible
Imagine being able to look inside a single cancer cell and see how it communicates with its neighbors. Scientists are celebrating a new technique that lets them study the fatty contents of cancer cells, [...]
Antibiotic Breakthrough: Revolutionary Chinese Study Paves Way for Superbug Defeating Drugs
New research reveals that fluorous lipopetides act as highly effective antibiotics. Bacterial infections resistant to multiple drugs, which no existing antibiotics can treat, represent a significant worldwide challenge. A research group from China has [...]
Signs of Multiple Sclerosis Show Up in Blood Years Before Symptoms Appear
UCSF scientists clear a potential path toward earlier treatment for a disease that affects nearly 1,000,000 people in the United States. By Levi Gadye In a discovery that could hasten treatment for patients with multiple [...]
Advanced RNA Sequencing Reveals the Drivers of New COVID Variants
A study reveals that a new sequencing technique, tARC-seq, can accurately track mutations in SARS-CoV-2, providing insights into the rapid evolution and variant development of the virus. The SARS-CoV-2 virus that causes COVID has the unsettling [...]
No More Endless Boosters? Scientists Develop One-for-All Virus Vaccine
End of the line for endless boosters? Researchers at UC Riverside have developed a new vaccine approach using RNA that is effective against any strain of a virus and can be used safely even by babies or the immunocompromised. Every [...]
How Are Hydrogels Shaping the Future of Biomedicine?
Hydrogels have gained widespread recognition and utilization in biomedical engineering, with their applications dating back to the 1960s when they were first used in contact lens production. Hydrogels are distinguished from other biomaterials in [...]
Nanovials method for immune cell screening uncovers receptors that target prostate cancer
A recent UCLA study demonstrates a new process for screening T cells, part of the body's natural defenses, for characteristics vital to the success of cell-based treatments. The method filters T cells based on [...]
New Research Reveals That Your Sense of Smell May Be Smarter Than You Think
A new study published in the Journal of Neuroscience indicates that the sense of smell is significantly influenced by cues from other senses, whereas the senses of sight and hearing are much less affected. A popular [...]
Deadly bacteria show thirst for human blood: the phenomenon of bacterial vampirism
Some of the world's deadliest bacteria seek out and feed on human blood, a newly-discovered phenomenon researchers are calling "bacterial vampirism." A team led by Washington State University researchers has found the bacteria are [...]
Organ Architects: The Remarkable Cells Shaping Our Development
Finding your way through the winding streets of certain cities can be a real challenge without a map. To orient ourselves, we rely on a variety of information, including digital maps on our phones, [...]
Novel hydrogel removes microplastics from water
Microplastics pose a great threat to human health. These tiny plastic debris can enter our bodies through the water we drink and increase the risk of illnesses. They are also an environmental hazard; found [...]
Researchers Discover New Origin of Deep Brain Waves
Understanding hippocampal activity could improve sleep and cognition therapies. Researchers from the University of California, Irvine’s biomedical engineering department have discovered a new origin for two essential brain waves—slow waves and sleep spindles—that are critical for [...]
The Lifelong Cost of Surviving COVID: Scientists Uncover Long-Term Effects
Many of the individuals released to long-term acute care facilities suffered from conditions that lasted for over a year. Researchers at UC San Francisco studied COVID-19 patients in the United States who survived some of the longest and [...]
Previously Unknown Rogue Immune Key to Chronic Viral Infections Discovered
Scientists discovered a previously unidentified rogue immune cell linked to poor antibody responses in chronic viral infections. Australian researchers have discovered a previously unknown rogue immune cell that can cause poor antibody responses in [...]
Nature’s Betrayal: Unmasking Lead Lurking in Herbal Medicine
A case of lead poisoning due to Ayurvedic medicine use demonstrates the importance of patient history in diagnosis and the need for public health collaboration to prevent similar risks. An article in CMAJ (Canadian Medical Association [...]
Frozen in Time: How a DNA Anomaly Misled Scientists for Centuries
An enormous meteor spelled doom for most dinosaurs 65 million years ago. But not all. In the aftermath of the extinction event, birds — technically dinosaurs themselves — flourished. Scientists have spent centuries trying [...]