Let's say I have a text classifier that I have trained on some training dataset. Now when I run this trained classifier on some test dataset, I identify the cases where it went wrong (assuming I have correct labels for test dataset as well).
How can I re-train the classifier such that it learns from its mistakes? Applies to both binary and multi-class setting.
PS: Since this is a broad question, any pointers to relevant articles would suffice as well.
Assuming you just automatically get the correct labels for the test set after making a prediction, this is a standard problem known as online learning. Just incorporate the new labels into your training set and update your model. For standard modern models learned via SGD, as long as you think your test set is approximately IID from the target distribution, you can just keep updating your model with SGD; this might risk catastrophic forgetting, however.
To answer your comment elsewhere:
How would that be any different from batch learning? I understand that online learning incrementally keeps updating the model but alternatively, I can also wait for some time to collect some test data and then retrain the model using all of data. Right?
Yes, this would be one strategy for "online learning." This is also fine, but more expensive than simply updating your model as you go (since you have to completely retrain, and store all of the data you've ever seen). It might make you slightly more robust to the test data not being completely IID, but it also might make you take longer to adapt to true concept drift. If you're using a complex model whose optimization surface is non-convex, e.g. a deep network, then you also might have a chance to reach a better solution by retraining with more data.
concept drift? Possibly because I will be using training data that is now stale? But the incremental learning also does not really mitigates that issue. Does it? Also can you please point me to methods that are commonly used to deal with concept drift? Thank you.
$\endgroup$
Feb 8, 2018 at 22:40
Cost-sensitive C5.0 decision tree ensembles implement a misclassification penalty. This method may be of interest. Check out Applied Predictive Modelling by Kuhn and Jonnson.
Boosting approach can be useful in this case. In boosting, each data sample gets weighted so that the misclassified obs. have a higher chance of being used for the next classifier.
Note: boosting is not a single model, it is an ensemble of many models, as many as you wish.
