Apple: ‘Reasoning’ AIs fail hard if they actually have to think
Anything outside their training.
Text version: https://pivot-to-ai.com/2025/06/08/ap...
Patreon: / davidgerard
Ko-Fi: https://ko-fi.com/A1529D5
Buy me nice things: https://www.amazon.co.uk/hz/wishlist/...
Get an extremely cool Pivot to AI shirt or mug: https://pivot-to-ai.redbubble.com
Sources:
The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (PDF) https://ml-site.cdn-apple.com/papers/...
GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (PDF) https://arxiv.org/pdf/2410.05229
Stop Anthropomorphizing Intermediate Tokens As Reasoning/Thinking Traces! (PDF) https://arxiv.org/pdf/2504.09762
Previously on Pivot to AI:
LLMs can solve any word problem! As long as they can crib the answer https://pivot-to-ai.com/2024/07/06/ll...
AI benchmarks are self-promoting trash — but regulators keep using them https://pivot-to-ai.com/2025/02/25/ai...
‘Reasoning’ AI is LYING to you! — or maybe it’s just hallucinating again https://pivot-to-ai.com/2025/04/18/re...
video: • ‘Reasoning’ AI is LYING to you! — or maybe...
Full Pivot to AI playlist: • Uploads from Pivot to AI
Audio-only podcast: https://pivottoai.libsyn.com