While GPT-4 performs well in structured reasoning tasks, a new study shows that its ability to adapt to variations is weak—suggesting AI still lacks true abstract understanding and flexibility in decision-making.
Artificial Intelligence (AI), particularly large language models like GPT-4, has shown impressive performance on reasoning tasks. But does AI truly understand abstract concepts, or is it just mimicking patterns? A new study from the University of Amsterdam and the Santa Fe Institute reveals that while GPT models perform well on some analogy tasks, they fall short when the problems are altered, highlighting key weaknesses in AI’s reasoning capabilities.
Analogical reasoning is the ability to draw a comparison between two different things based on their similarities in certain aspects. It is one of the most common methods by which human beings try to understand the world and make decisions. An example of analogical reasoning: cup is to coffee as soup is to ??? (the answer being: bowl)
Large language models like GPT-4 perform well on various tests, including those requiring analogical reasoning. But can AI models truly engage in general, robust reasoning, or do they over-rely on patterns from their training data? This study by language and AI experts Martha Lewis (Institute for Logic, Language and Computation at the University of Amsterdam) and Melanie Mitchell (Santa Fe Institute) examined whether GPT models are as flexible and robust as humans in making analogies. ‘This is crucial, as AI is increasingly used for decision-making and problem-solving in the real world,’ explains Lewis.
Comparing AI models to human performance
Lewis and Mitchell compared the performance of humans and GPT models on three different types of analogy problems:
- Letter sequences – Identify patterns in letter sequences and complete them correctly.
- Digit matrices – Analyzing number patterns and determining the missing numbers.
- Story analogies – Understanding which of two stories best corresponds to a given example story.
A system that truly understands analogies should maintain high performance even on variations
In addition to testing whether GPT models could solve the original problems, the study examined how well they performed when the problems were subtly modified. ‘A system that truly understands analogies should maintain high performance even on these variations’, state the authors in their article.
GPT models struggle with robustness
Humans maintained high performance on most modified versions of the problems, but GPT models, while performing well on standard analogy problems, struggled with variations. ‘This suggests that AI models often reason less flexibly than humans, and their reasoning is less about true abstract understanding and more about pattern matching,’ explains Lewis.
In digit matrices, GPT models showed a significant performance drop when the missing number’s position changed. Humans had no difficulty with this. In story analogies, GPT-4 tended to select the first given answer as correct more often, whereas humans were not influenced by answer order. Additionally, GPT-4 struggled more than humans when key elements of a story were reworded, suggesting a reliance on surface-level similarities rather than deeper causal reasoning.
When tested on modified versions, GPT models showed a decline in performance on simpler analogy tasks, while humans remained consistent. However, both humans and AI struggled with more complex analogical reasoning tasks.
Weaker than human cognition
This research challenges the widespread assumption that AI models like GPT-4 can reason in the same way humans do. ‘While AI models demonstrate impressive capabilities, this does not mean they truly understand what they are doing,’ conclude Lewis and Mitchell. ‘Their ability to generalize across variations is still significantly weaker than human cognition. GPT models often rely on superficial patterns rather than deep comprehension.’
This is a critical warning about using AI in important decision-making areas such as education, law, and healthcare. While AI can be a powerful tool, it is not yet a replacement for human thinking and reasoning.
- Lewis, Martha, and Melanie Mitchell. “Evaluating the Robustness of Analogical Reasoning in Large Language Models.” Transactions on Machine Learning Research, 2025, openreview.net/forum?id=t5cy5v9wp
News
Cancer’s Hidden Antioxidant Shield Helps It Escape the Immune System
Blocking an antioxidant protein that tumors use to suppress immune attacks improved cancer immunotherapy responses in mice. Cancer cells can release antioxidants that interfere with the immune cells trying to kill them. Researchers have [...]
Scientists Find a Berry Compound That Helps Muscle Cells Burn Fat
Pterostilbene, a compound found naturally in some foods, affects how skeletal muscles process fats by stabilizing a protein called PPARδ and increasing its signaling activity. Pterostilbene, a natural compound found in blueberries, grapes, and [...]
Two Hidden Forces Help Build the Human Brain Before Birth
Scientists have uncovered two surprising forces that help guide how the human brain forms before birth. Before birth, the human brain is shaped in large part by an unusual class of stem cells known [...]
Researchers Uncover a Hidden Trigger Behind Chronic Inflammation
The findings offer new insights that could help guide the development of future therapies. A protein called human resistin may help flip on one of the immune system’s most powerful inflammatory switches. Researchers at [...]
Scientists Have Uncovered Previously Hidden Microbial Activity on Human Skin
The most abundant microbes on your skin may not be the ones doing most of the work. Human skin supports vast communities of bacteria, fungi, and viruses that can influence its protective barrier, immune [...]
Brazilian Tree Compounds Fight COVID-19 on Multiple Fronts
Scientists found compounds in a Brazilian tree that hit SARS-CoV-2 on multiple fronts, revealing a promising new lead in the search for COVID-19 treatments. Researchers have found that galloylquinic acids extracted from the leaves [...]
Cutting Two Amino Acids Slowed Prostate Cancer in Mice
A newly identified link between amino acid metabolism and cholesterol production may help prostate tumors adapt to hormone therapy. Prostate cancer can find ways around treatments designed to deprive tumors of the hormones they [...]
Largest-Ever Physics Survey Raises New Doubts About Our Model of the Universe
Physicists around the world remain deeply divided on key mysteries of the universe, from dark matter to quantum gravity. The standard cosmological model failed to gain majority support, and no leading theory dominated the [...]
Scientists Just Overturned a 100-Year-Old Belief About Bacteria in the Lungs
New findings raise questions about the role of microbes living in the lungs. More than 35 trillion bacteria live throughout the human body, forming microbiomes in the gut, mouth, lungs, skin, and urogenital tract. [...]
GHCE Concept
From the preface of the book Global Health Care Equivalency in the Age of Nanotechnology, Nanomedicine and Artificial Intelligence, Edited by Frank Boehm: Since the publication of my first book (Nanomedical Device and Systems [...]
Novartis, Ionis drug failure spurs questions
Pelacarsen didn’t protect heart health despite lowering levels of a protein particle, “Lp(a),” in a large clinical trial — a result with important implications for cardiovascular drug research. Dive Brief: An RNA drug from [...]
New injectable treatment helps the brain rebuild after stroke
Biomedical engineers at Duke University have created an injectable biomaterial that may help the brain recover from damage left behind by an ischemic stroke. In experiments with mice, the material transformed the cavity created [...]
Scientists Discover a Hidden “Immune Organ” Inside the Skull
Researchers discovered lymph node-like immune hubs inside skull bone marrow that appear to act as rapid-response centers for the brain. For decades, the brain was thought to operate largely apart from the immune system. [...]
Engineered tRNAs and lipid nanoparticles target nonsense mutation cystic fibrosis
Researchers have developed a potential new approach for treating a form of cystic fibrosis caused by so-called nonsense mutations, combining chemically modified transfer RNAs with lipid nanoparticles designed to deliver the therapy directly to [...]
New pancreatic cancer drug carries a $39,800 monthly list price
A groundbreaking treatment for one of the most common forms of pancreatic cancer has been approved in pill form by the FDA. Revolution Medicines’ oral tablet daraxonrasib, branded as Rasonque, reduced the risk of [...]
Researchers Have Discovered a New Way To Reduce Chronic Nerve Pain
A cancer-linked protein called BRAF may help drive chronic nerve pain, and existing cancer drugs targeting it reduced pain sensitivity in preclinical models. Chronic nerve pain can persist long after an injury and often [...]
