Determine the most important documents for supervised learning

Question

I have somewhat of a general/high level question.

Assume I'm doing supervised machine learning on some text data (tweets for example) and categorizing the documents to a certain taxonomy (multi-class classification). My supervised model performs fairly well on testing data but what I'm trying to do now is find a way to sort out which documents from the production data (not human labeled) if added to the training set would improve accuracy the most on that particular production data set.

The idea is once the initial predictions are made on production data, another algorithm would be run to determine the n most important documents (say 500) that if manually labeled by a human and added to the training set would improve accuracy the most, assuming the model does a 2nd round of prediction after the n documents have been added to the training set.

So the process would look like this:

1. Initial predictions on production data are made
2. Algorithm runs to determine the most important records to increase accuracy of the model (based on features most likely).
3. Human reviews these most important records and adds them to the training set
4. A 2nd round of prediction takes place, hopefully with a better accuracy metric

I'm thinking of using something like K-Means after doing some dimensionality reduction since it's NLP problem with lots of features.

Does anyone have any experience or suggestions in regard to this topic? Am I on the right track?

Answer 1

Perhaps you might find it useful to take cues from query strategy frameworks in active learning, which evaluate the informativeness of unlabeled instances (so as to pick instances to be labeled). A good starting resource could be Section 3 of Active Learning Literature Survey (2010).

For example, the uncertainty sampling strategy means picking the labels for which your model's predictions have the highest uncertainty for---if your model has some sort of entropy measure.

As for what the process might look like, a possible alternative to selecting these ~500 important records at once might be to iteratively select, possibly in batches, query instances to be labeled by an oracle. This would allow later query instance choices to be better-informed.