ModeLing: A Novel Dataset for Testing Linguistic Reasoning in Language Models
ModeLing: A Novel Dataset for Testing Linguistic Reasoning in Language Models
Authors: Nathan Andrew Chi, Teodor Malchev, Riley Kong, Ryan Andrew Chi, Lucas Huang, Ethan A Chi, R. Thomas McCoy and Dragomir Radev
Abstract: Towards understanding the capability of models to perform multilingual few-shot reasoning, we propose MODELING , a benchmark of Rosetta stone puzzles (Bozhanov and Derzhanski, 2013). This type of puzzle, originating from competitions called Linguistics Olympiads, contain a small number of sentences in a target language not previously known to the solver. Each sentence is translated
to the solver’s language such that the provided sentence pairs uniquely specify a single most reasonable underlying set of rules; solving requires applying these rules to translate new expressions.
#SIGTYP2024