1
$\begingroup$

Let's say I have a text classifier that I have trained on some training dataset. Now when I run this trained classifier on some test dataset, I identify the cases where it went wrong (assuming I have correct labels for test dataset as well).

How can I re-train the classifier such that it learns from its mistakes? Applies to both binary and multi-class setting.

PS: Since this is a broad question, any pointers to relevant articles would suffice as well.

$\endgroup$
1
$\begingroup$

Assuming you just automatically get the correct labels for the test set after making a prediction, this is a standard problem known as online learning. Just incorporate the new labels into your training set and update your model. For standard modern models learned via SGD, as long as you think your test set is approximately IID from the target distribution, you can just keep updating your model with SGD; this might risk catastrophic forgetting, however.

To answer your comment elsewhere:

How would that be any different from batch learning? I understand that online learning incrementally keeps updating the model but alternatively, I can also wait for some time to collect some test data and then retrain the model using all of data. Right?

Yes, this would be one strategy for "online learning." This is also fine, but more expensive than simply updating your model as you go (since you have to completely retrain, and store all of the data you've ever seen). It might make you slightly more robust to the test data not being completely IID, but it also might make you take longer to adapt to true concept drift. If you're using a complex model whose optimization surface is non-convex, e.g. a deep network, then you also might have a chance to reach a better solution by retraining with more data.

$\endgroup$
  • $\begingroup$ Thank you so much Dougal. Your answers have been very helpful and provided the exact information that I was looking for. Just one last follow up: Why would it take me longer to adapt to concept drift? Possibly because I will be using training data that is now stale? But the incremental learning also does not really mitigates that issue. Does it? Also can you please point me to methods that are commonly used to deal with concept drift? Thank you. $\endgroup$ – The Wanderer Feb 8 '18 at 22:40
  • $\begingroup$ Yeah, just because you'll still be actively using old training data. If you use a large enough learning rate in SGD, and especially if your model is convex, then eventually you'll forget about the old stuff with an online setup, whereas it'll always contribute to your loss (though less and less) with the retraining model. I don't know a ton about online learning in particular, though. $\endgroup$ – Dougal Feb 8 '18 at 23:22
0
$\begingroup$

Cost-sensitive C5.0 decision tree ensembles implement a misclassification penalty. This method may be of interest. Check out Applied Predictive Modelling by Kuhn and Jonnson.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.