A computer-assisted needle misses its target, puncturing the spine. A diabetic patient goes rapidly downhill after a computer recommends an incorrect insulin dosage. An ultrasound fails to diagnose an obvious heart condition that is ultimately fatal.
These are just a few examples of incidents reported to the United States’ Food and Drug Administration involving health technology assisted by artificial intelligence (AI), and Australian researchers say they are an “early warning sign” of what could happen if regulators, hospitals and patients don’t take safety seriously in the rapidly evolving field.
“This is essentially showing us that when we’re putting in AI systems, we just need to be taking the safety of these systems really seriously,” Professor Farah Magrabi said.
Her team at the Australian Institute of Health Innovation at Macquarie University this month published a review of 266 safety events involving AI-assisted technology reported to the US watchdog. The article appeared in the Journal of the American Medical Informatics Association.
Only 16 per cent actually led to patients being harmed, but two-thirds were found to have the potential to cause harm, and 4 per cent were categorised as “near-miss events” in which users intervened.
Co-author Dr David Lyell said issues arose most commonly when users failed to enter the correct data, leading to an incorrect result, or misunderstood what the AI was actually telling them when it produced a result.
For example, one patient suffering a heart attack delayed medical care because an over-the-counter electrocardiogram device – which is not capable of detecting a heart attack – told them they had “normal sinus rhythm”.
“AI isn’t the answer; it’s part of a system that needs to support the provision of healthcare. And we do need to make sure that we have the systems in place that supports its effective use to promote healthcare for people,” Lyell said.
The researchers chose to analyse cases in the US, where the implementation of AI-enabled health devices is more advanced than in Australia. The US regulator has, to date, approved 521 artificial intelligence and machine learning-enabled medical devices, with 178 of those added in 2022.
In Australia, the Therapeutic Goods Administration (TGA) does not collect data about the number of approved devices in Australia that have AI or machine-learning components, but Magrabi said the regulator was taking the issue “very, very seriously”.
“AI isn’t the answer, it’s part of a system that needs to support the provision of health care.”
David Lyell, Australian Institute of Health Innovation, Macquarie University
AI has been used in healthcare devices for decades, but an explosion in data collection, computing power and advanced algorithms has opened up new frontiers, said David Hansen, chief executive of the CSIRO’s Australian E-Health Research Centre.
Sydney-based medical device start-up EMVision is one Australian company taking advantage of these advancements to develop a portable device for diagnosing stroke without the need for an MRI.
In the development stage, the company is using an advanced algorithm and high-powered computers to simulate stroke in numerous places in the brain, building up a database of synthetic images similar to MRIs which are then compared with real-life MRI and CT results from hospital clinical trials at Royal Melbourne, Liverpool and Princess Alexandra hospitals.
“We wouldn’t be able to do what we’re doing today if we didn’t have the [high-powered computer] infrastructure for the simulation,” said head of product development Forough Khandan.
Co-founder Scott Kirkland said the intention was not to completely replace CT and MRI scans, but to diagnose stroke in the first hour when treatment is most effective. The bedside device, set to be launched in 2025, will use a “traffic light” system based on a probability algorithm to help determine what type of stroke might have occurred.
“It’s better for an algorithm to give an ‘I don’t know’ than an incorrect answer, and have the wrong treatment and or triage process followed,” Kirkland said.
Radiology is at the forefront of the rapid adoption of AI in healthcare, especially in breast cancer screening and analysis of chest X-rays.
“A couple of years ago, almost no radiologist would say they use it, now a fair percentage would say that they use it in their daily work,” said clinical radiologist and AI safety researcher Dr Lauren Oakden Rayner.
Rayner, a member of the college, said the technology had many potential benefits, but Australian regulators and clinicians needed to better understand the risks of fully autonomous systems before putting them into hospitals, clinics and homes.
“Humans are legally and morally responsible for decision-making, and it’s taking some of that out of human hands,” she said. “There’s no reason autonomous AI systems can’t exist … but they obviously have to be tested very, very tightly.”
News
AI therapy may help with mental health, but innovation should never outpace ethics
Mental health services around the world are stretched thinner than ever. Long wait times, barriers to accessing care and rising rates of depression and anxiety have made it harder for people to get timely help. As a result, governments and health care providers are [...]
Global life expectancy plunges as WHO warns of deepening health crisis Post-COVID
The World Health Organization (WHO) has sounded the alarm on the long-term health repercussions of the COVID-19 pandemic in its newly released World Health Statistics Report 2025. The report reveals a staggering decline in global [...]
Researchers map brain networks involved in word retrieval
How are we able to recall a word we want to say? This basic ability, called word retrieval, is often compromised in patients with brain damage. Interestingly, many patients who can name words they [...]
Melting Ice Is Changing the Color of the Ocean – Scientists Are Alarmed
Melting sea ice changes not only how much light enters the ocean, but also its color, disrupting marine photosynthesis and altering Arctic ecosystems in subtle but profound ways. As global warming causes sea ice in the [...]
Your Washing Machine Might Be Helping Antibiotic-Resistant Bacteria Spread
A new study reveals that biofilms in washing machines may contain potential pathogens and antibiotic resistance genes, posing possible risks for laundering healthcare workers’ uniforms at home. Washing healthcare uniforms at home could be [...]
Scientists Discover Hidden Cause of Alzheimer’s Hiding in Plain Sight
Researchers found the PHGDH gene directly causes Alzheimer’s and discovered a drug-like molecule, NCT-503, that may help treat the disease early by targeting the gene’s hidden function. A recent study has revealed that a gene previously [...]
How Brain Cells Talk: Inside the Complex Language of the Human Mind
Introduction The human brain contains nearly 86 billion neurons, constantly exchanging messages like an immense social media network, but neurons do not work alone – glial cells, neurotransmitters, receptors, and other molecules form a vast [...]
Oxford study reveals how COVID-19 vaccines prevent severe illness
A landmark study by scientists at the University of Oxford, has unveiled crucial insights into the way that COVID-19 vaccines mitigate severe illness in those who have been vaccinated. Despite the global success of [...]
Annual blood test could detect cancer earlier and save lives
A single blood test, designed to pick up chemical signals indicative of the presence of many different types of cancer, could potentially thwart progression to advanced disease while the malignancy is still at an early [...]
How the FDA opens the door to risky chemicals in America’s food supply
Lining the shelves of American supermarkets are food products with chemicals linked to health concerns. To a great extent, the FDA allows food companies to determine for themselves whether their ingredients and additives are [...]
Superbug crisis could get worse, killing nearly 40 million people by 2050
The number of lives lost around the world due to infections that are resistant to the medications intended to treat them could increase nearly 70% by 2050, a new study projects, further showing the [...]
How Can Nanomaterials Be Programmed for Different Applications?
Nanomaterials are no longer just small—they are becoming smart. Across fields like medicine, electronics, energy, and materials science, researchers are now programming nanomaterials to behave in intentional, responsive ways. These advanced materials are designed [...]
Microplastics Are Invading Our Arteries, and It Could Be Increasing Your Risk of Stroke
Higher levels of micronanoplastics were found in carotid artery plaque, especially in people with stroke symptoms, suggesting a potential new risk factor. People with plaque buildup in the arteries of their neck have been [...]
Gene-editing therapy shows early success in fighting advanced gastrointestinal cancers
Researchers at the University of Minnesota have completed a first-in-human clinical trial testing a CRISPR/Cas9 gene-editing technique to help the immune system fight advanced gastrointestinal (GI) cancers. The results, recently published in The Lancet Oncology, show encouraging [...]
Engineered extracellular vesicles facilitate delivery of advanced medicines
Graphic abstract of the development of VEDIC and VFIC systems for high efficiency intracellular protein delivery in vitro and in vivo. Credit: Nature Communications (2025). DOI: 10.1038/s41467-025-59377-y. https://www.nature.com/articles/s41467-025-59377-y Researchers at Karolinska Institutet have developed a technique [...]
Brain-computer interface allows paralyzed users to customize their sense of touch
University of Pittsburgh School of Medicine scientists are one step closer to developing a brain-computer interface, or BCI, that allows people with tetraplegia to restore their lost sense of touch. While exploring a digitally [...]
