A computer-assisted needle misses its target, puncturing the spine. A diabetic patient goes rapidly downhill after a computer recommends an incorrect insulin dosage. An ultrasound fails to diagnose an obvious heart condition that is ultimately fatal.
These are just a few examples of incidents reported to the United States’ Food and Drug Administration involving health technology assisted by artificial intelligence (AI), and Australian researchers say they are an “early warning sign” of what could happen if regulators, hospitals and patients don’t take safety seriously in the rapidly evolving field.
“This is essentially showing us that when we’re putting in AI systems, we just need to be taking the safety of these systems really seriously,” Professor Farah Magrabi said.
Her team at the Australian Institute of Health Innovation at Macquarie University this month published a review of 266 safety events involving AI-assisted technology reported to the US watchdog. The article appeared in the Journal of the American Medical Informatics Association.
Only 16 per cent actually led to patients being harmed, but two-thirds were found to have the potential to cause harm, and 4 per cent were categorised as “near-miss events” in which users intervened.
Co-author Dr David Lyell said issues arose most commonly when users failed to enter the correct data, leading to an incorrect result, or misunderstood what the AI was actually telling them when it produced a result.
For example, one patient suffering a heart attack delayed medical care because an over-the-counter electrocardiogram device – which is not capable of detecting a heart attack – told them they had “normal sinus rhythm”.
“AI isn’t the answer; it’s part of a system that needs to support the provision of healthcare. And we do need to make sure that we have the systems in place that supports its effective use to promote healthcare for people,” Lyell said.
The researchers chose to analyse cases in the US, where the implementation of AI-enabled health devices is more advanced than in Australia. The US regulator has, to date, approved 521 artificial intelligence and machine learning-enabled medical devices, with 178 of those added in 2022.
In Australia, the Therapeutic Goods Administration (TGA) does not collect data about the number of approved devices in Australia that have AI or machine-learning components, but Magrabi said the regulator was taking the issue “very, very seriously”.
“AI isn’t the answer, it’s part of a system that needs to support the provision of health care.”
David Lyell, Australian Institute of Health Innovation, Macquarie University
AI has been used in healthcare devices for decades, but an explosion in data collection, computing power and advanced algorithms has opened up new frontiers, said David Hansen, chief executive of the CSIRO’s Australian E-Health Research Centre.
Sydney-based medical device start-up EMVision is one Australian company taking advantage of these advancements to develop a portable device for diagnosing stroke without the need for an MRI.
In the development stage, the company is using an advanced algorithm and high-powered computers to simulate stroke in numerous places in the brain, building up a database of synthetic images similar to MRIs which are then compared with real-life MRI and CT results from hospital clinical trials at Royal Melbourne, Liverpool and Princess Alexandra hospitals.
“We wouldn’t be able to do what we’re doing today if we didn’t have the [high-powered computer] infrastructure for the simulation,” said head of product development Forough Khandan.
Co-founder Scott Kirkland said the intention was not to completely replace CT and MRI scans, but to diagnose stroke in the first hour when treatment is most effective. The bedside device, set to be launched in 2025, will use a “traffic light” system based on a probability algorithm to help determine what type of stroke might have occurred.
“It’s better for an algorithm to give an ‘I don’t know’ than an incorrect answer, and have the wrong treatment and or triage process followed,” Kirkland said.
Radiology is at the forefront of the rapid adoption of AI in healthcare, especially in breast cancer screening and analysis of chest X-rays.
“A couple of years ago, almost no radiologist would say they use it, now a fair percentage would say that they use it in their daily work,” said clinical radiologist and AI safety researcher Dr Lauren Oakden Rayner.
Rayner, a member of the college, said the technology had many potential benefits, but Australian regulators and clinicians needed to better understand the risks of fully autonomous systems before putting them into hospitals, clinics and homes.
“Humans are legally and morally responsible for decision-making, and it’s taking some of that out of human hands,” she said. “There’s no reason autonomous AI systems can’t exist … but they obviously have to be tested very, very tightly.”

News
The Surprising Origin of a Deadly Hospital Infection
C. diff might not originate from external transmission but rather from within the infected patient themselves. Hospital staff dedicate significant effort to safeguard patients from infections during their hospital stay. Through practices ranging from [...]
Google AI breakthrough – huge step in finding genes that cause diseases
Google says it has made a significant step in identifying disease-causing genes, which could help spot rare genetic disorders. A new model named AlphaMissense is able to confidently classify 89 per cent of all [...]
New Study: Everyday Pleasures Can Boost Cognitive Performance
MINDWATCH study reveals cognitive peaks with everyday pleasures. Listening to music and drinking coffee are the sorts of everyday pleasures that can impact a person’s brain activity in ways that improve cognitive performance, including [...]
Moderna reveals new highly targeted COVID-19 vaccine mRNA-1283
Moderna has developed a new and improved version of its COVID-19 vaccine. The unique formulation (mRNA-1283) reduces the vaccine's content from the full-length SARS-CoV-2 spike protein to a narrowly focused encoding of just two [...]
New nanotech weapon takes aim at hard-to-treat breast cancer
Breast cancer in its various forms affects more than 250,000 Americans a year. One particularly aggressive and hard-to-treat type is triple-negative breast cancer (TNBC), which lacks specific receptors targeted by existing treatments. The rapid [...]
Scientists upcycle plastics into liquids that can store hydrogen energy
Scientists from Nanyang Technological University, Singapore (NTU Singapore) have created a process that can upcycle most plastics into chemical ingredients useful for energy storage, using light-emitting diodes (LEDs) and a commercially available catalyst, all [...]
Yale Scientists Uncover How the Immune System Can Alter Our Behavior
The mere scent of seafood can severely sicken those allergic to it — and therefore they are more likely to avoid it. Similarly, individuals who experience food poisoning from a specific dish tend to [...]
Whirlwind Tech – The Future of Energy-Efficient Spintronics Computing
Researchers in Germany and Japan have been able to increase the diffusion of magnetic whirls, so-called skyrmions, by a factor of ten. In today’s world, our lives are unimaginable without computers. Up until now, [...]
Omicron’s Silver Lining: Significantly Lower Risk of Long COVID
Omicron infections have a lower risk of long COVID than earlier variants, according to a study analyzing data from 11,000 participants. The risk of developing long COVID is significantly lower following an infection with [...]
The Hidden Mechanism Connecting Diabetes and Cancer
Researchers have discovered that insulin resistance, typically linked with type 2 diabetes, is also present in cancer patients and can accelerate the spread of the disease. In the 1920s, scientists found that the urine [...]
Scientists Unveil Urea’s Secret Role in the Origin of Life
Scientists from ETH Zurich and the University of Geneva have developed a new technique that allows them to observe chemical reactions taking place in liquids at extremely high temporal resolution. This innovation enables them to track how molecules [...]
Viagra Lowers Alzheimer’s Risk by Almost 70%, Early Study Finds
Research published recently suggests that Pfizer’s erectile dysfunction drug Viagra can decrease the risk of developing Alzheimer’s disease by up to 69 percent. The research, which was published in Nature, found that the medication has [...]
Future of Medical Imaging: Advanced AI Can Tell Your True Age by Looking at Your Chest
An AI-powered model utilizes chest X-rays to help develop biomarkers for aging. What if determining “your age” was based on your chest rather than your face? Scientists from Osaka Metropolitan University have crafted an [...]
Ultra-sensitive biosensors detect cancer in a blood test
Cancer biomarkers circulating in body fluids can be used for diagnosis and treatment monitoring. However, current detection technology lacks the required sensitivity, limiting biomarker use in clinical applications. Colorectal cancer is the second most [...]
Viruses cause 200+ diseases. This one drug may be able to treat them all.
By taking aim at a process common across many viruses, the drug could one day stop any number of known viruses — and new ones. t’s about as audacious an idea as you can [...]
Scientists Identify Potential Treatment for Rare and Devastating Lung Disease
The findings could lead to a cure for LAM. Researchers from the University of Cincinnati may have identified a potential treatment for lymphangioleiomyomatosis (LAM), a rare lung condition resembling cancer found predominantly in women of reproductive [...]