In this case desire is to optionally label a final test set after training. The training is to be done using an active learning approach which can be biased towards different characteristics (indecision, confidence, etc).

Would the test set in this scenario be useful at all? For example, if the active learning algorithm was biased to select high confidence samples the remaining data would have a different mix of data than what was started with (less positives and less high confidence data). Would totally invalidate the test set or can you still draw valid conclusions from it?

  • $\begingroup$ You should measure your predictions against your test set, so it should be labeled. You perform active learning in order to choose which unlabelled samples to labelled. Since the test set is already labeled (and should be used for testing) the active learning should be done on a separate unlabelled data set. $\endgroup$
    – DaL
    Feb 28 '17 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.