When covid-19 struck Europe in March 2020, hospitals were plunged into a health crisis that was still badly understood. “Doctors really didn’t have a clue how to manage these patients,” says Laure Wynants, an epidemiologist at Maastricht University in the Netherlands, who studies predictive tools.
But there was data coming out of China, which had a four-month head start in the race to beat the pandemic. If machine-learning algorithms could be trained on that data to help doctors understand what they were seeing and make decisions, it just might save lives. “I thought, ‘If there’s any time that AI could prove its usefulness, it’s now,’” says Wynants. “I had my hopes up.”
It never happened—but not for lack of effort. Research teams around the world stepped up to help. The AI community, in particular, rushed to develop software that many believed would allow hospitals to diagnose or triage patients faster, bringing much-needed support to the front lines—in theory.
In the end, many hundreds of predictive tools were developed. None of them made a real difference, and some were potentially harmful.
That’s the damning conclusion of multiple studies published in the last few months. In June, the Turing Institute, the UK’s national center for data science and AI, put out a report summing up discussions at a series of workshops it held in late 2020. The clear consensus was that AI tools had made little, if any, impact in the fight against covid.
Not fit for clinical use
This echoes the results of two major studies that assessed hundreds of predictive tools developed last year. Wynants is lead author of one of them, a review in the British Medical Journal that is still being updated as new tools are released and existing ones tested. She and her colleagues have looked at 232 algorithms for diagnosing patients or predicting how sick those with the disease might get. They found that none of them were fit for clinical use. Just two have been singled out as being promising enough for future testing.
“It’s shocking,” says Wynants. “I went into it with some worries, but this exceeded my fears.”
Wynants’s study is backed up by another large review carried out by Derek Driggs, a machine-learning researcher at the University of Cambridge, and his colleagues, and published in Nature Machine Intelligence. This team zoomed in on deep-learning models for diagnosing covid and predicting patient risk from medical images, such as chest x-rays and chest computer tomography (CT) scans. They looked at 415 published tools and, like Wynants and her colleagues, concluded that none were fit for clinical use.
“This pandemic was a big test for AI and medicine,” says Driggs, who is himself working on a machine-learning tool to help doctors during the pandemic. “It would have gone a long way to getting the public on our side,” he says. “But I don’t think we passed that test.”
Both teams found that researchers repeated the same basic errors in the way they trained or tested their tools. Incorrect assumptions about the data often meant that the trained models did not work as claimed.
Wynants and Driggs still believe AI has the potential to help. But they are concerned that it could be harmful if built in the wrong way because they could miss diagnoses or underestimate risk for vulnerable patients. “There is a lot of hype about machine-learning models and what they can do today,” says Driggs.
Unrealistic expectations encourage the use of these tools before they are ready. Wynants and Driggs both say that a few of the algorithms they looked at have already been used in hospitals, and some are being marketed by private developers. “I fear that they may have harmed patients,” says Wynants.
So what went wrong? And how do we bridge that gap? If there’s an upside, it is that the pandemic has made it clear to many researchers that the way AI tools are built needs to change. “The pandemic has put problems in the spotlight that we’ve been dragging along for some time,” says Wynants.
What went wrong
Many of the problems that were uncovered are linked to the poor quality of the data that researchers used to develop their tools. Information about covid patients, including medical scans, was collected and shared in the middle of a global pandemic, often by the doctors struggling to treat those patients. Researchers wanted to help quickly, and these were the only public data sets available. But this meant that many tools were built using mislabeled data or data from unknown sources.
A group of researchers proposed an optimized electroforming strategy to achieve a bioinspired nano-holed TiO2 coated Ti6Al4V alloy, according to a study published in the journal ACS Applied Material Interfaces. Near-infrared (NIR) energy plays a critical part [...]
The Centers for Disease Control and Prevention estimates that more than 2.8 million Americans experience antibiotic-resistant infections each year; more than 35,000 die from those infections. To address this critical and worldwide public health [...]
Scientists at The University of Texas MD Anderson Cancer Center and Northwestern Medicine have identified natural extracellular vesicles containing the ACE2 protein (evACE2) in the blood of COVID-19 patients that can block infection from [...]
Scientists at Northwestern Medicine and The University of Texas MD Anderson Cancer Center have identified natural nano-bubbles containing the ACE2 protein (evACE2) in the blood of COVID-19 patients and discovered these nano-sized particles can [...]
Pioneering Artificial Intelligence (AI) technology, developed by experts at University of the West of Scotland (UWS), is capable of accurately diagnosing Covid-19 in just a few minutes. The groundbreaking program is able to detect [...]
Antimicrobial resistance poses a significant threat to humanity, health leaders have warned, as a study reveals it has become a leading cause of death worldwide and is killing about 3,500 people every day. More [...]
The Necessity of Novel Antiviral Materials With the severe acute respiratory coronavirus 2 (SARS-CoV-2) being highly infectious and transmissible, the need for novel antiviral materials that can decrease viral activities has become paramount. Preparation procedures [...]
Individuals living with Type 1 diabetes must carefully follow prescribed insulin regimens every day, receiving injections of the hormone via syringe, insulin pump or some other device. And without viable long-term treatments, this course [...]
Researchers from the University of Sydney Nano Institute and School of Chemistry have revealed that tiny gas bubbles – nanobubbles just 100 billionths of a metre high – form on surfaces in unexpected situations, [...]
Innovative researchers have investigated the potential of incorporating a gelatin methacryloyl hydrogel functionalized with synthetic nanoclay laponite to improve the delivery of osteoblast derived extracellular vesicles for increased bone repair. This research has been [...]
A couple of essential biomolecules play a key role in the process of blood clotting. One of these is a protein called fibrinogen. It is often given to those who have experienced heavy bleeding, [...]
Nanoemulsions are a relatively new technology that has found significant use for delivering functional chemicals such as micronutrients, flavorings, bioactive molecules, and antimicrobial agents into food and beverage products. This article focuses on applying [...]
Researchers at Penn Medicine have discovered a new, more effective method of preventing the body's own proteins from treating nanomedicines like foreign invaders, by covering the nanoparticles with a coating to suppress the immune [...]
Bioinspired by tunicates and mussels, a Korean research team has created a “bomb-like” anticancer therapeutic agent that only destroys cancer cells. Schematic diagram of the mechanism of action of photoactivatable adhesive nanobombs made with [...]
(Reuters) - “Long COVID”, where symptoms of COVID-19 persist for months after an initial infection, could be emerging as a chronic disease in Finland, Minister of Family Affairs and Social Services Krista Kiuru said [...]