Say we have unlimited unlabeled data and we can ask an oracle for labels. We can use active learning to choose the most informative data samples for labeling, thus minimizing data labeling cost. If we choose a single sample at a time, we can choose the one with the highest class entropy.

How do we go about choosing a batch of k samples to be labeled, e.g. for deep learning?

Some approaches I encountered are:

  1. Choose k samples with the highest class entropy.
  2. As in 1., but ensure that there are the same amount of samples from every class in the batch.
  3. Select K samples with the highest class entropy and randomly choose k samples out of them.

Are there any more sophisticated approaches that work in every scenario? Which of the above would be the best to use?


I can recommend you Yuxin Chen and Andreas Krause's NIPS'13 paper about Near-optimal Batch Mode Active Learning and Adaptive Submodular Optimization.

Reformulating your hypothesis tests using the Adaptive Submodularity framework provides great theoretical run-time guarantees for greedy algorithms.

In this paper, they showed that their hit-and-run sampler works near-optimal for sampling for any results which are confined by a convex body (like linear separators). For a non convex body, you can use hit-and-run as a good heuristic.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.