Consider the situation where we have a trained classifier (which isn't dramatically over/underfitted) that we want to improve, and lots of unlabeled data readily available, and we would like to spend some (finite) resources to get them classified. An example of this is scraping social media posts and then manually analyzing them (for example classifying whether a post is offensive or not). There are virtually unlimited posts, but you need to spend time/money to manually classify them.
I'm looking for methods of determining which unlabeled samples should be manually labeled, but I'm unable to find any papers on this problem (possibly because I don't know what this problem is named).
So far, I came up with 3 options (well, actually 2, since the first one is what happens if you don't even think about the problem):
- Just picking samples at random, to better mirror their "natural" occurence - but this seems like a waste, especially if the data is inherently imbalanced (in my case it is),
- Running unlabeled samples through my classifier and picking the ones with probability closest to 0.5 (i.e. the ones where the classifier was the most unsure) - but there is a risk of the classifier never fixing its past mistakes (if it's sure about labeling something incorrectly, this sample will not be included in the dataset),
- Running unlabeled samples through my classifier and sampling with weights based on probability density function of my classifier's predicted probabilities of normal distribution centered at 0.5 - this should in theory counteract the problems of the second approach.
So far I'm inclined towards the last option, but I am sure I'm not the first person who has to solve this problem, so there probably are lots of papers on this issue that I am unable to find.
I would be happy to be pointed towards any sources on the subject, or to have my ideas criticized.