Do Large Reasoning Models Truly Think Like Humans? Apple’s AI team Analysis : A Deep Dive into Their Capabilities

Large Reasoning Models (LRMs) have been hailed as a potential leap toward artificial general intelligence (AGI), with their ability to generate detailed "thought processes" before providing answers. Industry leaders like OpenAI’s o1/o3, DeepSeek-R1, Claude 3.7 Sonnet Thinking, and Gemini Thinking have positioned these models as frontrunners in tackling complex problems. But do they genuinely possess human-like reasoning, or are they just sophisticated pattern-matchers? A recent study by Apple’s machine learning research team puts these models to the test, revealing surprising insights into their strengths and limitations.

A New Approach to Testing AI Reasoning

Traditional benchmarks like MATH-500, AIME24, and AIME25 are commonly used to evaluate reasoning models, but they often suffer from "data contamination," where models may memorize answers rather than truly understand problems. To address this, Apple’s research team designed a novel testing framework using controlled intellectual puzzles, including:

• Tower of Hanoi: A classic problem involving moving disks between three pegs.
• Checkers Puzzle: Swapping colored pieces on a linear board.
• River-Crossing Puzzle: Variants of the classic "wolf, sheep, cabbage, and farmer" problem with complex rules.
• Block World: Rearranging lettered blocks to achieve a target configuration.

These puzzles allow researchers to precisely adjust difficulty (e.g., increasing the number of disks in Tower of Hanoi) and analyze the models’ thought processes, as they are unlikely to have been pre-exposed to these specific scenarios.

Key Findings: The Limits of AI Reasoning

The study uncovered several critical insights into the performance of large reasoning models:

1. Performance Varies by Problem Complexity

Models showed distinct performance patterns across different levels of problem complexity:

• Low Complexity: Non-reasoning models often outperformed reasoning models in terms of efficiency and accuracy.
• Medium Complexity: Reasoning models began to shine, demonstrating a clear advantage over non-reasoning counterparts.
• High Complexity: Both reasoning and non-reasoning models experienced a complete performance collapse, with accuracy dropping to zero.

2. Counterintuitive Reasoning Effort

Surprisingly, the amount of computational effort (measured in tokens used during reasoning) didn’t always increase with problem difficulty. As complexity rose, models initially consumed more tokens, but near the threshold of performance collapse, they paradoxically reduced their reasoning effort. This was particularly evident in the o3-mini model, while Claude-3.7-Sonnet (Thinking) showed less pronounced behavior.

Researchers suggest this indicates a fundamental scalability limit in current reasoning models, where they struggle to cope with highly complex problems.

3. Inconsistent Reasoning Processes

By analyzing the reasoning trajectories, the team found that models often overthought simple problems, identifying correct solutions early but then exploring incorrect paths, wasting resources. For moderately complex problems, models initially pursued wrong solutions before correcting themselves later in the process. In high-complexity scenarios, they failed to generate any correct solutions, resulting in zero accuracy.

4. Puzzling Behaviors and Instability

In a surprising experiment, researchers provided models with a complete recursive algorithm for the Tower of Hanoi puzzle, expecting better performance since executing a known algorithm should be simpler than deriving one. However, the models still collapsed under high-difficulty scenarios, suggesting that their limitations extend beyond finding solutions to executing precise instructions.

Additionally, model performance was inconsistent across puzzle types. For instance, models could execute hundreds of correct steps in Tower of Hanoi before erring, but they often failed within just a few steps in river-crossing puzzles, despite the latter requiring fewer optimal steps. This inconsistency may stem from sparse training data for certain puzzle types, like river-crossing problems.

Implications for the Future of AI Reasoning

While the study focused on a narrow slice of reasoning tasks (puzzle-based problems), it raises significant questions about the capabilities of large reasoning models. The findings suggest that these models have a clear performance ceiling, and simply scaling up data or computational power may not overcome these limitations. Their ability to generalize reasoning across diverse problem types appears limited, and their reliance on pattern-matching rather than true reasoning remains a valid concern.

The road to AGI may be longer than anticipated. While large reasoning models demonstrate impressive capabilities, their inability to handle high-complexity problems and inconsistent performance across tasks highlight the need for new approaches to AI development.

Apple’s research provides a fresh perspective on the state of large reasoning models, challenging the hype surrounding their human-like thinking abilities. By using carefully designed intellectual puzzles, the study reveals both the strengths and critical weaknesses of these models. As the AI community continues to push toward more advanced systems, understanding these limitations will be crucial for developing truly intelligent machines.