Comparing Active Learning Algorithms on Toy-Datasets: A Critical Evaluation

In this research paper, the authors aim to address the limited availability of large-scale datasets for adversarial learning (AL) research. They identify three main challenges in existing AL studies: (1) most datasets are small and cannot be used with larger models, (2) there is a lack of diverse datasets across different domains and class sizes, and (3) the gap between state-of-the-art algorithms and random sampling is often insignificant. To address these challenges, the authors introduce a set of vector datasets that are solvable by medium-sized models in under 1000 samples, have a significant gap between algorithms and random sampling, and include multiple data domains and class sizes. The authors use Splice, DNA, and USPS from LibSVMTools as their selected vector datasets.
The authors also analyze the existing AL research on text datasets, finding that most papers use either News Category or TopV2, despite these datasets receiving less attention in AL studies. They argue that these datasets are "toy-datasets" and may not be relevant for practical purposes but still provide valuable insights into the performance of different AL algorithms.
To further compare the performance of various AL algorithms on different datasets, the authors propose a new experimental setting based on the work of [8]. They offer all their datasets in both raw and pre-encoded forms using a fixed embedding model trained by unsupervised contrastive learning.
The authors’ findings demonstrate that their proposed datasets provide a more comprehensive and diverse benchmark for AL research, enabling the evaluation of different algorithms on various data types and sizes. By making these datasets publicly available, the authors hope to encourage more research on AL and improve its practical applications.
In conclusion, this paper tackles the limitations of existing AL research by creating a more comprehensive and diverse benchmark for evaluating different algorithms. By providing a set of vector datasets that can be used with medium-sized models, the authors aim to promote further research on AL and its practical applications in various domains.