Perception vs. Comprehension:Exploring the Mathematical Reasoning Boundaries of LLMs
Evaluating mathematical reasoning in Large Language Models (LLMs) using benchmarks like GSM8K can be limited. The GSM-Symbolic benchmark addresses this by enabling researchers to explore variations in questions. Researchers have found that LLMs perform differently depending on question wording and struggle with complex questions, suggesting that they may be recognizing patterns rather than understanding mathematical concepts. The GSM-NoOp dataset further demonstrates the limitations of LLMs in discerning relevant information, highlighting their reliance on pattern matching. These findings indicate that LLMs may not genuinely grasp mathematical principles and require improvements in reasoning abilities.