Iman Mirzadeh, Keivan Alizadeh-Vahid, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, Mehrdad Farajtabar
GSM-Symbolic is a benchmark designed to more rigorously evaluate the mathematical reasoning abilities of LLMs, revealing the limitations of existing GSM8K and showing that models rely on pattern matching rather than genuine logical reasoning.
Although LLM performance on GSM8K has greatly improved, it is unclear whether this reflects genuine advances in mathematical reasoning or learning of data biases. Existing evaluations fail to measure robustness to problem variations, thus not providing reliable metrics.
GSM-Symbolic uses symbolic templates to automatically generate problems with varying numbers and conditions. This enables controlled experiments, such as changing only numerical values or adding irrelevant clauses, to precisely analyze models' reasoning capabilities.
All state-of-the-art models showed significant performance variance when numerical values were changed, and performance dropped by up to 65% when irrelevant clauses were added. This strongly suggests that LLMs rely on surface-level patterns in training data rather than true logical reasoning, providing important insights for future model evaluation and improvement.