I have a bag of words binary text classification task. The SGD algorithm performed well for a certain target where number of labeled cases for training reached tens of thousands. For another target only about a thousand of labeled cases is available.
Colleagues can help me to manually label any additional case to the new target. Thus the "assisted" learning comes to mind. Number of unlabeled cases we have is virtually unlimited. The limitation is that colleagues can only label a few hundreds of cases at most so we have to choose them wisely.
How to chose cases to label with most impact to the precision of the classifier?
I recall reading a text online about choosing the cases with the highest uncertainty of classification or something but can't find it now.
Could you point me to any algorithm or a just some starting point to implement the above approach?