Evaluating Inductive Reasoning Capabilities of Large Language Models With The One Dimensional Abstract Reasoning Corpus

Abstract

We present an initial automated test to evaluate the LLMs’ capacity to perform inductive reasoning tasks. We use the GPT-3.5/4 models to create a system which generates Python code as hypotheses for inductive reasoning to transform sequences of the One Dimensional Abstract Reasoning Corpus (1D-ARC) challenge. We experiment with 3 prompting techniques, namely standard prompting, Chain of Thought (CoT) and direct feedback. We provide results and an analysis of cost to success rate and benefit-cost ratio. Our best result is an overall 25% success rate with our CoT prompting on GPT-4, significantly surpass- ing the standard prompting approach. We discuss potential avenues to improve our experiments and test other strategies.