How can I interpret the results of this training process?

Question

I trained a classifier that outputs probabilities for 3 classes. I produced the plot of the weighted accuracy achieved on test and train set, throughout training.

First, I want to explain how I constructed the contingency table to check if I did a reasonable thing. For each instance, I picked as predicted label the one with the highest probability, and considered it as true prediction when it was equal to the true label.

I understand that from just before epoch 20 onwards the model is kind of converged but there is also a little bit of overfitting (is that right?). But how can I interpret what happened before that, also talking about underfitting and good fit?

Answer 1

First, I want to explain how I constructed the contingency table to check if I did a reasonable thing. For each instance, I picked as predicted label the one with the highest probability, and considered it as true prediction when it was equal to the true label.

This is a perfectly reasonable method for testing how well models predict class membership when they're only allowed to output a class. If you want to evaluate the probabilities produced by models, though, you need a proper scoring rule. This is the difference between ordinary classification and probabilistic classification. You can use some of the same models for both of these tasks, but they're different tasks.

Your graph suggests the classifier is indeed underfit at lower numbers of epochs, since neither training accuracy nor test accuracy have reached their potential yet. But I don't see overfitting at higher number of epochs. Although training accuracy is higher than test accuracy there, test accuracy isn't decreasing as training accuracy increases—that would be overfitting. Rather, both training and testing accuracies seem to be increasing a little bit.

Answer 2

I don't see any sign for overfitting here, at least not from the graph. Your training and test performance seem to go in parallel. This is generally a good thing because your classifier is able to generalize whatever you learn from your data onto something it has never seen before.

How you did the classification is very common and make total sense.