I have somewhat of a general/high level question.
Assume I'm doing supervised machine learning on some text data (tweets for example) and categorizing the documents to a certain taxonomy (multi-class classification). My supervised model performs fairly well on testing data but what I'm trying to do now is find a way to sort out which documents from the production data (not human labeled) if added to the training set would improve accuracy the most on that particular production data set.
The idea is once the initial predictions are made on production data, another algorithm would be run to determine the n most important documents (say 500) that if manually labeled by a human and added to the training set would improve accuracy the most, assuming the model does a 2nd round of prediction after the n documents have been added to the training set.
So the process would look like this:
- Initial predictions on production data are made
- Algorithm runs to determine the most important records to increase accuracy of the model (based on features most likely).
- Human reviews these most important records and adds them to the training set
- A 2nd round of prediction takes place, hopefully with a better accuracy metric
I'm thinking of using something like K-Means after doing some dimensionality reduction since it's NLP problem with lots of features.
Does anyone have any experience or suggestions in regard to this topic? Am I on the right track?