My training accuracy is 1.0 and my test accuracy is 0.994. Am I overfitting for a multiclass classification?

Question

This is a multiclass classification for an imbalanced dataset. I set the class_weight for this model to "balanced". I have a perfect training accuracy (1.0) and a nearly perfect testing accuracy (0.994). I looked at my confusion matrices but they predicted each class really well. Am I overfitting? I ran a cross val score on the features and targets before train test split, and I got a cross val score of 0.996.

Training confusion matrix:

array([[1., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0.],
       [0., 0., 1., 0., 0.],
       [0., 0., 0., 1., 0.],
       [0., 0., 0., 0., 1.]])

Testing confusion matrix:

array([[0.997, 0.003 , 0.    , 0., 0.],
       [0.   , 1.    , 0.    , 0., 0.],
       [0.   , 0.    , 1.    , 0., 0.],
       [0.   , 0.    , 0.    , 1., 0.],
       [0.01 , 0.    , 0.007, 0., 0.980]])

Answer 1

You're right to take a critical view of the test score. However, if the test set was correctly chosen, you are not overfitting.

I ran a cross val score on the features and targets before train test split

If you ran the cross validation and any feature extraction on ALL the data before train-test splitting, then what you may have done is accidentally incorporated information from the test set into your model. I've made that mistake before.

You can test your model by running the train test split before anything else, and use a specified random state so you can make it reproducible, and try your model again.

Answer 2

One simple way to check if your model is overfitting is by plotting the training/validation losses/accuracies along epochs.

If your training loss curve does not decrease or your acc curve does not increase, than your model are not able to fit the training data. You can try to incrase the model capacity, change optimization algorithm, etc.

If your training loss curve decrease or acc curve increase, but the validation ones start to go the different direction. Then your model may overfit to the trianing data. You can mitigate the overfitting by adding regularization, data augmentation, etc.

Finally, when the training and validation losses both go down, it is good.

One caveat is the when you observe your model have low validation accuracy, it is not always "overfitting". You should always investigate training loss first.

Answer 3

A quick check is to ask if you are able to outperform a baseline model that predicts the majority category every time. You mention imbalance. Depending on how imbalanced your data are, you might be able to get better than $99.4\%$ accuracy by just predicting the majority category every time, such as if one category accounts for $99.9\%$ of the cases and you predict that category every time. This would scream out overfitting, as moving from a simple model (predict one category every time) to a complex model (whatever you do) results in a rise in the in-sample performance yet a drop in out-of-sample performance.