# The Illusion of Thinking:Understanding Strengths Limitations of Reasoning Models Source: https://machinelearning.apple.com/research/illusion-of-thinking Kiwi News: https://news.kiwistand.com/stories/The-Illusion-of-ThinkingUnderstanding-Strengths-Limitations-of-Reasoning-Models?index=0x68462137fcb9eec3cb1b7e7fbe309a5d07f29f6037923a6a789fa9072f9b1c91fb4e96e4 Submitted by misha on 2025-06-08 UTC, 3 upvotes, 1 comment ## Summary Apple researchers Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio and Mehrdad Farajtabar argue that frontier Large Reasoning Models (LRMs) suffer a complete accuracy collapse beyond certain problem complexities, despite better benchmark results. Using controllable puzzle environments, they compare LRMs with standard language models under equivalent inference compute and identify three regimes: standard models outperform on low-complexity tasks, LRMs gain an advantage on medium-complexity tasks, and both collapse on high-complexity tasks. They also report that LRMs' reasoning effort rises with complexity up to a point, then declines even with an adequate token budget, and that the models fail to use explicit algorithms and reason inconsistently across puzzles. ## Article AuthorsParshin Shojaee*†, Iman Mirzadeh*, Keivan Alizadeh, Maxwell Horton, Samy Bengio, Mehrdad Farajtabar Recent generations of frontier language models have introduced Large Reasoning Models (LRMs) that generate detailed thinking processes before providing answers. While these models demonstrate improved performance on reasoning benchmarks, their fundamental capabilities, scaling properties, and limitations remain insufficiently understood. Current evaluations primarily focus on established mathematical and coding benchmarks, emphasizing final answer accuracy. However, this evaluation paradigm often suffers from data contamination and does not provide insights into the reasoning traces’ structure and quality. In this work, we systematically investigate these gaps with the help of controllable puzzle environments that allow precise manipulation of compositional complexity while maintaining consistent logical structures. This setup enables the analysis of not only final answers but also the internal reasoning traces, offering insights into how LRMs “think”. Through extensive experimentation across diverse puzzles, we show that frontier LRMs face a complete accuracy collapse beyond certain complexities. Moreover, they exhibit a counter-intuitive scaling limit: their reasoning effort increases with problem complexity up to a point, then declines despite having an adequate token budget. By comparing LRMs with their standard LLM counterparts under equivalent inference compute, we identify three performance regimes: (1) low-complexity tasks where standard models surprisingly outperform LRMs, (2) medium-complexity tasks where additional thinking in LRMs demonstrates advantage, and (3) high-complexity tasks where both models experience complete collapse. We found that LRMs have limitations in exact computation: they fail to use explicit algorithms and reason inconsistently across puzzles. We also investigate the reasoning traces in more depth, studying the patterns of explored solutions and analyzing the models’ computational behavior, shedding light on their strengths, limitations, and ultimately raising crucial questions about their true reasoning capabilities. * Equal contribution † Work done while at Apple Related readings and updates. Apple researchers advance AI and ML through fundamental research, and to support the broader research community and help accelerate progress in this field, we share much of this work through publications and engagement at conferences. Next month, the 39th annual Conference on Neural Information Processing Systems (NeurIPS), will be held in San Diego, California, with a satellite event also taking place in Mexico City,… Read more We introduce TASER (Translation Assessment via Systematic Evaluation and Reasoning), a metric that uses Large Reasoning Models (LRMs) for automated translation quality assessment. TASER harnesses the explicit reasoning capabilities of LRMs to conduct systematic, step-by-step evaluation of translation quality. We evaluate TASER on the WMT24 Metrics Shared Task across both reference-based and reference-free scenarios, demonstrating state-of-the-art… Read more ## Comments **misha**: I asked Claude to explain this paper to a 5-year old: Imagine you have a really smart robot friend who tries to solve puzzles. The robot can look at easy puzzles and say "Hmm, let me think about this..." and solve them pretty well! It even looks like it's really thinking hard, just like you do when you solve puzzles. But here's the trick: When you give the robot really hard puzzles, it gets stuck. Even though it still says "Let me think..." and looks like it's thinking super hard, it can't actually figure out the answer. It's like if you could do 2+2, but when someone asks you to do really big math, you just can't - even if you pretend to think really hard about it. The robot is the same way - it looks like it's thinking, but it can only solve easy stuff, not the really hard stuff. So the scientists discovered that these robot friends aren't as smart as we thought - they're just really good at pretending to think!