Measuring Models’ Ability to Falsehood: A Comprehensive Evaluation of Truthfulness in Deep Learning

In this article, researchers propose a new benchmark called FIND to evaluate the interpretability of machine learning models, specifically Large Language Models (LLMs). The goal is to assess how well these models can describe and explain their inner workings. FIND consists of two main tasks: a function description task and an extrapolation task.
The function description task requires the LLMs to provide clear and concise explanations for a given function, similar to a human teacher describing a mathematical concept to a student. The researchers created a dataset of functions with different levels of complexity to test the models’ ability to generalize and abstract underlying rules.
In the extrapolation task, the models are presented with unseen functions and must generate new examples based on the patterns and structures observed in the original dataset. This task evaluates the LLMs’ capacity to generate creative and innovative solutions beyond what they have seen before.
The authors of this paper argue that current evaluation standards for LLMs, such as language proficiency tests or simple math problems, are no longer sufficient as these models continue to improve in capabilities. FIND aims to address this issue by providing a more comprehensive assessment of LLMs’ interpretability and their ability to describe complex functions and processes.
To create the FIND dataset, researchers drew from existing benchmarks for spatial understanding and reasoning, such as the Map Navigation Task (MNT) and the Driving Directions Task (DDT). These tasks require models to process and understand spatial information, similar to how humans do in everyday life. By combining these tasks with the function description and extrapolation tasks, FIND offers a more comprehensive evaluation of LLMs’ abilities.
In summary, FIND is a new benchmark designed to assess the interpretability of LLMs by testing their ability to describe and explain complex functions and processes. The dataset consists of two main tasks: function description and extrapolation, and it aims to provide a more comprehensive evaluation of these models’ capabilities beyond language proficiency tests or simple math problems.