Large-scale datasets (L2D) are critical in human-in-the-loop decision making, as they provide the necessary fuel for training and evaluating machine learning models. However, these datasets often lack robust benchmarking, which hinders research into this field. This survey aims to address this issue by formalizing how to define capacity constraints and providing insights into various approaches used in L2D.
Defining Capacity Constraints
Humans have limitations when processing instances, such as the number of instances they can handle in a given time period. In real-world systems, these constraints must be applied over batches of instances, not individual datasets. To represent capacity constraints, we use a vector that indicates which batch an instance belongs to and a human capacity matrix that defines how many instances each expert can process within a given time period.
Approaches to L2D
Several approaches have been proposed for L2D, including (1) using small datasets with limited human predictions, (2) synthesizing expert behavior using simplistic methods, and (3) leveraging large-scale datasets containing diverse examples. These approaches vary significantly in their complexity and performance, but they all aim to improve the efficiency and accuracy of machine learning models.
Benchmarking L2D
Robust benchmarking is crucial for evaluating the effectiveness of L2D methods. However, this task is challenging due to the lack of standardized datasets and evaluation protocols. To address this issue, we propose a framework that defines capacity constraints and provides insights into the strengths and weaknesses of different L2D approaches.
Conclusion
L2D has the potential to revolutionize human-in-the-loop decision making by providing more accurate and efficient machine learning models. However, there are still significant challenges to overcome before these methods can be widely adopted. By formalizing capacity constraints and benchmarking L2D approaches, we can better understand their strengths and weaknesses and improve their performance in real-world applications.
