The most recent version of ChatGPT, an AI chatbot developed for language interpretation and response generation, has successfully passed a radiology board-style exam, demonstrating both its potential and limitations, according to research studies published in the Radiological Society of North America’s journal.
The latest version of ChatGPT passed a radiology board-style exam, highlighting the potential of large language models but also revealing limitations that hinder reliability, according to two new research studies published in Radiology, a journal of the Radiological Society of North America (RSNA).
ChatGPT is an artificial intelligence (AI) chatbot that uses a deep learning model to recognize patterns and relationships between words in its vast training data to generate human-like responses based on a prompt. But since there is no source of truth in its training data, the tool can generate responses that are factually incorrect.
ChatGPT was recently named the fastest growing consumer application in history, and similar chatbots are being incorporated into popular search engines like Google and Bing that physicians and patients use to search for medical information, Dr. Bhayana noted.
To assess its performance on radiology board exam questions and explore strengths and limitations, Dr. Bhayana and colleagues first tested ChatGPT based on GPT-3.5, currently the most commonly used version. The researchers used 150 multiple-choice questions designed to match the style, content and difficulty of the Canadian Royal College and American Board of Radiology exams.
The questions did not include images and were grouped by question type to gain insight into performance: lower-order (knowledge recall, basic understanding) and higher-order (apply, analyze, synthesize) thinking. The higher-order thinking questions were further subclassified by type (description of imaging findings, clinical management, calculation and classification, disease associations).
The performance of ChatGPT was evaluated overall and by question type and topic. Confidence of language in responses was also assessed.
The researchers found that ChatGPT based on GPT-3.5 answered 69% of questions correctly (104 of 150), near the passing grade of 70% used by the Royal College in Canada. The model performed relatively well on questions requiring lower-order thinking (84%, 51 of 61), but struggled with questions involving higher-order thinking (60%, 53 of 89). More specifically, it struggled with higher-order questions involving description of imaging findings (61%, 28 of 46), calculation and classification (25%, 2 of 8), and application of concepts (30%, 3 of 10). Its poor performance on higher-order thinking questions was not surprising given its lack of radiology-specific pretraining.
GPT-4 was released in March 2023 in limited form to paid users, specifically claiming to have improved advanced reasoning capabilities over GPT-3.5.
In a follow-up study, GPT-4 answered 81% (121 of 150) of the same questions correctly, outperforming GPT-3.5 and exceeding the passing threshold of 70%. GPT-4 performed much better than GPT-3.5 on higher-order thinking questions (81%), more specifically those involving description of imaging findings (85%) and application of concepts (90%).
The findings suggest that GPT-4’s claimed improved advanced reasoning capabilities translate to enhanced performance in a radiology context. They also suggest improved contextual understanding of radiology-specific terminology, including imaging descriptions, which is critical to enable future downstream applications.
“Our study demonstrates an impressive improvement in performance of ChatGPT in radiology over a short time period, highlighting the growing potential of large language models in this context,” Dr. Bhayana said.
GPT-4 showed no improvement on lower-order thinking questions (80% vs 84%) and answered 12 questions incorrectly that GPT-3.5 answered correctly, raising questions related to its reliability for information gathering.
“We were initially surprised by ChatGPT’s accurate and confident answers to some challenging radiology questions, but then equally surprised by some very illogical and inaccurate assertions,” Dr. Bhayana said. “Of course, given how these models work, the inaccurate responses should not be particularly surprising.”
ChatGPT’s dangerous tendency to produce inaccurate responses, termed hallucinations, is less frequent in GPT-4 but still limits usability in medical education and practice at present.
Both studies showed that ChatGPT used confident language consistently, even when incorrect. This is particularly dangerous if solely relied on for information, Dr. Bhayana notes, especially for novices who may not recognize confident incorrect responses as inaccurate.
“To me, this is its biggest limitation. At present, ChatGPT is best used to spark ideas, help start the medical writing process and in data summarization. If used for quick information recall, it always needs to be fact-checked,” Dr. Bhayana said.
News
A Virus Designed in the Lab Could Help Defeat Antibiotic Resistance
Scientists can now design bacteria-killing viruses from DNA, opening a faster path to fighting superbugs. Bacteriophages have been used as treatments for bacterial infections for more than a century. Interest in these viruses is rising [...]
Sleep Deprivation Triggers a Strange Brain Cleanup
When you don’t sleep enough, your brain may clean itself at the exact moment you need it to think. Most people recognize the sensation. After a night of inadequate sleep, staying focused becomes harder [...]
Lab-grown corticospinal neurons offer new models for ALS and spinal injuries
Researchers have developed a way to grow a highly specialized subset of brain nerve cells that are involved in motor neuron disease and damaged in spinal injuries. Their study, published today in eLife as the final [...]
Urgent warning over deadly ‘brain swelling’ virus amid fears it could spread globally
Airports across Asia have been put on high alert after India confirmed two cases of the deadly Nipah virus in the state of West Bengal over the past month. Thailand, Nepal and Vietnam are among the [...]
This Vaccine Stops Bird Flu Before It Reaches the Lungs
A new nasal spray vaccine could stop bird flu at the door — blocking infection, reducing spread, and helping head off the next pandemic. Since first appearing in the United States in 2014, H5N1 [...]
These two viruses may become the next public health threats, scientists say
Two emerging pathogens with animal origins—influenza D virus and canine coronavirus—have so far been quietly flying under the radar, but researchers warn conditions are ripe for the viruses to spread more widely among humans. [...]
COVID-19 viral fragments shown to target and kill specific immune cells
COVID-19 viral fragments shown to target and kill specific immune cells in UCLA-led study Clues about extreme cases and omicron’s effects come from a cross-disciplinary international research team New research shows that after the [...]
Smaller Than a Grain of Salt: Engineers Create the World’s Tiniest Wireless Brain Implant
A salt-grain-sized neural implant can record and transmit brain activity wirelessly for extended periods. Researchers at Cornell University, working with collaborators, have created an extremely small neural implant that can sit on a grain of [...]
Scientists Develop a New Way To See Inside the Human Body Using 3D Color Imaging
A newly developed imaging method blends ultrasound and photoacoustics to capture both tissue structure and blood-vessel function in 3D. By blending two powerful imaging methods, researchers from Caltech and USC have developed a new way to [...]
Brain waves could help paralyzed patients move again
People with spinal cord injuries often lose the ability to move their arms or legs. In many cases, the nerves in the limbs remain healthy, and the brain continues to function normally. The loss of [...]
Scientists Discover a New “Cleanup Hub” Inside the Human Brain
A newly identified lymphatic drainage pathway along the middle meningeal artery reveals how the human brain clears waste. How does the brain clear away waste? This task is handled by the brain’s lymphatic drainage [...]
New Drug Slashes Dangerous Blood Fats by Nearly 40% in First Human Trial
Scientists have found a way to fine-tune a central fat-control pathway in the liver, reducing harmful blood triglycerides while preserving beneficial cholesterol functions. When we eat, the body turns surplus calories into molecules called [...]
A Simple Brain Scan May Help Restore Movement After Paralysis
A brain cap and smart algorithms may one day help paralyzed patients turn thought into movement—no surgery required. People with spinal cord injuries often experience partial or complete loss of movement in their arms [...]
Plant Discovery Could Transform How Medicines Are Made
Scientists have uncovered an unexpected way plants make powerful chemicals, revealing hidden biological connections that could transform how medicines are discovered and produced. Plants produce protective chemicals called alkaloids as part of their natural [...]
Scientists Develop IV Therapy That Repairs the Brain After Stroke
New nanomaterial passes the blood-brain barrier to reduce damaging inflammation after the most common form of stroke. When someone experiences a stroke, doctors must quickly restore blood flow to the brain to prevent death. [...]
Analyzing Darwin’s specimens without opening 200-year-old jars
Scientists have successfully analyzed Charles Darwin's original specimens from his HMS Beagle voyage (1831 to 1836) to the Galapagos Islands. Remarkably, the specimens have been analyzed without opening their 200-year-old preservation jars. Examining 46 [...]
