LabelBench: A Framework for Benchmarking Label-Efficient Learning

About this resource

Labeled data are critical to modern machine learning applications, but obtaining labels can be expensive. To mitigate this cost, machine learning methods, such as transfer learning, semi-supervised learning and active learning, aim to be label-efficient: achieving high predictive performance from relatively few labeled examples. While obtaining the best label-efficiency in practice often requires combinations of these techniques, existing benchmark and evaluation frameworks do not capture a concerted combination of all such techniques. This paper addresses this deficiency by introducing LabelBench, a new computationally-efficient framework for joint evaluation of multiple label-efficient learning techniques. As an application of LabelBench, we introduce a novel benchmark of state-of-the-art active learning methods in combination with semi-supervised learning for fine-tuning pretrained vision transformers. Our benchmark demonstrates better label-efficiencies than previously reported in active learning. LabelBench’s modular codebase is open-sourced for the broader community to contribute label-efficient learning methods and benchmarks.

Speaker: Jifan Zhang (https://jifanz.github.io/)is a PhD student at UW-Madison working with Prof. Robert Nowak. His work focuses on label-efficient learning and its modern application to large-scale deep learning systems. He is also generally interested in human-in-the-loop learning and cross-disciplinary research that apply these methods in the real world.

Project GitHub: https://github.com/EfficientTraining/LabelBench

See also

• Learn: video: Theoretical Foundations of Active Machine Learning: Learn more about the fundamentals of active learning, presented by Prof. Rob Nowak.