While AI models can break down problems into structured steps, new research reveals they still fail at basic arithmetic and fact-checking—raising questions about their true reasoning abilities.
Large Language Models (LLMs) have become indispensable in natural language processing, excelling at tasks such as sentiment analysis, reading comprehension, and answering factual questions. However, their ability to perform complex, multi-step reasoning remains a significant challenge, particularly in question-answering tasks that demand logical inference rather than simple recall. This study, authored by Nick Ferguson, Liane Guillou, Alan Bundy, and Kwabena Nuamah from the University of Edinburgh and Aveni, examines the extent to which LLMs can engage in two distinct forms of reasoning: meta-level and object-level reasoning.
Understanding Meta-Level and Object-Level Reasoning
Meta-level reasoning involves high-level strategic thinking, including problem decomposition and the formulation of intermediate steps necessary to solve a question. Object-level reasoning, in contrast, refers to the execution of these steps, such as performing mathematical calculations, retrieving specific facts, or applying symbolic logic. To evaluate the capabilities of LLMs in these areas, the authors introduce FRANKLIN, a novel dataset that explicitly requires models to engage in both reasoning types. FRANKLIN is inspired by the FRANK system, a symbolic reasoning framework for question answering, and focuses on geopolitical indicators such as population trends, economic metrics, and regional comparisons. Alongside three established multi-step question-answering datasets, FRANKLIN serves as a benchmark for testing the performance of four specific LLM versions: Meta’s Llama 3.1 8B, Microsoft’s Phi 3.5 Mini, Google’s Gemma 2 9B, and OpenAI’s GPT-4o-mini. Through two human annotation studies, the researchers assess whether LLMs can successfully generate reasoned responses and whether prompting them to plan their answers before execution improves their performance.
How LLMs Approach Reasoning Tasks
The study situates its analysis within the broader context of LLM reasoning tasks. As a cognitive function, reasoning encompasses logical deduction, belief revision, and inference-making. Common sense reasoning requires an understanding of everyday concepts and the ability to infer implicit knowledge. Mathematical reasoning demands numerical operations and logical problem-solving, while symbolic reasoning involves rule-based manipulations, such as emulating formal logic or deducing relationships between abstract entities. Multi-step reasoning is particularly significant, as it necessitates the sequential application of inference processes to arrive at a final answer. Despite their advancements, LLMs often struggle with these tasks because they rely on statistical pattern-matching rather than genuine logical deduction.
Existing techniques attempt to improve LLM performance on reasoning tasks. Fine-tuning involves additional training on domain-specific datasets to enhance accuracy in particular tasks while prompting techniques such as Chain-of-Thought (CoT) to introduce explicit reasoning steps into model responses. These approaches have demonstrated improvements, yet doubts remain as to whether LLMs are genuinely reasoning or merely imitating structured thought patterns learned from their training data. The authors propose a more structured classification of LLM reasoning, distinguishing between meta-level and object-level processes. While meta-level reasoning involves planning, selecting relevant knowledge sources, and determining the steps required to solve a problem, object-level reasoning focuses on accurate execution, including factual retrieval, numerical precision, and logical deductions.
FRANKLIN Dataset: A New Challenge for LLMs
To assess these reasoning types, the study introduces the FRANKLIN dataset, inspired by the FRANK system, which employs explicit symbolic reasoning to solve complex questions. FRANKLIN consists of complex questions requiring both meta- and object-level reasoning, particularly in the domain of geopolitical indicators. It includes scenarios requiring future prediction, regional comparisons, historical trends, and projections. Unlike more straightforward fact-retrieval datasets, FRANKLIN forces LLMs to not only determine the correct problem-solving approach but also accurately retrieve and manipulate relevant data. Each question is paired with a detailed explanation outlining the necessary reasoning steps. This dataset poses a significant challenge for LLMs, as it requires them not only to determine the appropriate strategy for answering a question but also to accurately retrieve and manipulate data.
How LLMs Were Evaluated: Two Human Annotation Studies
The evaluation design consists of two human annotation studies. In the first, LLMs were prompted to directly answer questions, allowing assessment of their object-level reasoning abilities. In the second, models were first asked to generate a plan before executing their reasoning steps, testing their meta-level reasoning skills. Participants rated responses based on their coherence, correctness, and the presence of structured reasoning. The study also introduced three key evaluation metrics:
- Answer Failure Rate (AFR) – the percentage of cases where an LLM provided no attempted answer.
- Rational Approach Rate (RAR) – the proportion of responses that outlined a coherent problem-solving approach.
- Plan Creation Rate (PCR) – the percentage of responses that structured their reasoning in a clear, step-by-step manner.
The results reveal a clear divergence in LLM performance between these two reasoning levels.
Key Findings: Meta-Level Strength, Object-Level Weakness
Across all datasets, LLMs consistently demonstrated strong meta-level reasoning. Responses often contained structured, step-by-step explanations that human annotators rated as rational and interpretable. Even for complex questions in FRANKLIN, models exhibited an ability to break down problems into intermediate steps and articulate a plan for solving them. However, while these responses appeared structured, the study raises concerns about whether they represent true reasoning or simply an imitation of learned patterns.
In contrast, LLMs struggled significantly with object-level reasoning. Object-level reasoning failures were frequent, particularly when questions required numerical precision or factual recall. In FRANKLIN, for example, models frequently fabricated numerical data, provided incorrect values, or made basic arithmetic errors. Even when models successfully identified the correct reasoning path, they often failed to follow through with accurate computations or fact retrieval. Error patterns included:
- Fabricating numerical data (e.g., citing non-existent sources).
- Retrieving inaccurate or imprecise information (e.g., rounding values incorrectly).
- Performing incorrect calculations (even for simple arithmetic operations).
A closer analysis of errors highlights the nature of these failures. Some responses contained entirely fabricated data, where models cited non-existent sources or invented statistical figures. Others retrieved information with reduced precision, rounding values or omitting key details necessary for accurate comparisons. In mathematical tasks, models often produce incorrect calculations, even for simple operations. These findings suggest that while LLMs can structure their responses in a way that appears logical, they lack the robust execution skills necessary to reliably generate correct answers in domains requiring object-level reasoning.
Implications for LLM Development
The findings have significant implications for the development of LLMs. While prompting models to engage in meta-level reasoning improves their ability to articulate coherent strategies, it does not address their deficiencies in object-level reasoning. This suggests that future advancements must focus on integrating external symbolic reasoning components, improving factual retrieval mechanisms, and refining numerical processing capabilities. The FRANKLIN dataset serves as a critical benchmark, demonstrating that even models with strong problem-decomposition skills struggle with execution.
Conclusion: The Path Forward for AI Reasoning
In conclusion, the study highlights a critical distinction in the reasoning capabilities of LLMs. While they can effectively plan and structure problem-solving approaches, their ability to execute complex reasoning tasks remains limited. The study’s findings emphasize that LLMs are proficient at mimicking reasoning structures but not necessarily reasoning in a human-like, cognitive sense. The introduction of FRANKLIN offers a new means of evaluating these deficiencies, laying the groundwork for further research into improving LLM performance in multi-step question answering. The results underscore the need for continued refinement in how LLMs handle object-level reasoning, ensuring that future iterations can move beyond surface-level imitation and towards genuine cognitive reasoning abilities.
- Preliminary scientific report. Ferguson, N., Guillou, L., Bundy, A., & Nuamah, K. (2025). Evaluating the Meta- and Object-Level Reasoning of Large Language Models for Question Answering. ArXiv. https://arxiv.org/abs/2502.10338

News
Unlocking hidden soil microbes for new antibiotics
Most bacteria cannot be cultured in the lab-and that's been bad news for medicine. Many of our frontline antibiotics originated from microbes, yet as antibiotic resistance spreads and drug pipelines run dry, the soil [...]
By working together, cells can extend their senses beyond their direct environment
The story of the princess and the pea evokes an image of a highly sensitive young royal woman so refined, she can sense a pea under a stack of mattresses. When it comes to [...]
Overworked Brain Cells May Hold the Key to Parkinson’s
Scientists at Gladstone Institutes uncovered a surprising reason why dopamine-producing neurons, crucial for smooth body movements, die in Parkinson’s disease. In mice, when these neurons were kept overactive for weeks, they began to falter, [...]
Old tires find new life: Rubber particles strengthen superhydrophobic coatings against corrosion
Development of highly robust superhydrophobic anti-corrosion coating using recycled tire rubber particles. Superhydrophobic materials offer a strategy for developing marine anti-corrosion materials due to their low solid-liquid contact area and low surface energy. However, [...]
This implant could soon allow you to read minds
Mind reading: Long a science fiction fantasy, today an increasingly concrete scientific goal. Researchers at Stanford University have succeeded in decoding internal language in real time thanks to a brain implant and artificial intelligence. [...]
A New Weapon Against Cancer: Cold Plasma Destroys Hidden Tumor Cells
Cold plasma penetrates deep into tumors and attacks cancer cells. Short-lived molecules were identified as key drivers. Scientists at the Leibniz Institute for Plasma Science and Technology (INP), working with colleagues from Greifswald University Hospital and [...]
This Common Sleep Aid May Also Protect Your Brain From Alzheimer’s
Lemborexant and similar sleep medications show potential for treating tau-related disorders, including Alzheimer’s disease. New research from Washington University School of Medicine in St. Louis shows that a commonly used sleep medication can restore normal sleep patterns and [...]
Sugar-Coated Nanoparticles Boost Cancer Drug Efficacy
A team of researchers at the University of Mississippi has discovered that coating cancer treatment carrying nanoparticles in a sugar-like material increases their treatment efficacy. They reported their findings in Advanced Healthcare Materials. Over a tenth of breast [...]
Nanoparticle-Based Vaccine Shows Promise in Fighting Cancer
In a study published in OncoImmunology, researchers from the German Cancer Research Center and Heidelberg University have created a therapeutic vaccine that mobilizes the immune system to target cancer cells. The researchers demonstrated that virus peptides combined [...]
Quantitative imaging method reveals how cells rapidly sort and transport lipids
Lipids are difficult to detect with light microscopy. Using a new chemical labeling strategy, a Dresden-based team led by André Nadler at the Max Planck Institute of Molecular Cell Biology and Genetics (MPI-CBG) and [...]
Ancient DNA reveals cause of world’s first recorded pandemic
Scientists have confirmed that the Justinian Plague, the world’s first recorded pandemic, was caused by Yersinia pestis, the same bacterium behind the Black Death. Dating back some 1,500 years and long described in historical texts but [...]
“AI Is Not Intelligent at All” – Expert Warns of Worldwide Threat to Human Dignity
Opaque AI systems risk undermining human rights and dignity. Global cooperation is needed to ensure protection. The rise of artificial intelligence (AI) has changed how people interact, but it also poses a global risk to human [...]
Nanomotors: Where Are They Now?
First introduced in 2004, nanomotors have steadily advanced from a scientific curiosity to a practical technology with wide-ranging applications. This article explores the key developments, recent innovations, and major uses of nanomotors today. A [...]
Study Finds 95% of Tested Beers Contain Toxic “Forever Chemicals”
Researchers found PFAS in 95% of tested beers, with the highest levels linked to contaminated local water sources. Per- and polyfluoroalkyl substances (PFAS), better known as forever chemicals, are gaining notoriety for their ability [...]
Long COVID Symptoms Are Closer To A Stroke Or Parkinson’s Disease Than Fatigue
When most people get sick with COVID-19 today, they think of it as a brief illness, similar to a cold. However, for a large number of people, the illness doesn't end there. The World [...]
The world’s first AI Hospital, developed in China is transforming healthcare
Artificial Intelligence and its developments have had a revolutionary impact on society, and healthcare is not an exception. China has made massive strides in AI integrated healthcare, and continues to do so as AI [...]