1
$\begingroup$

This is a multiclass classification for an imbalanced dataset. I set the class_weight for this model to "balanced". I have a perfect training accuracy (1.0) and a nearly perfect testing accuracy (0.994). I looked at my confusion matrices but they predicted each class really well. Am I overfitting? I ran a cross val score on the features and targets before train test split, and I got a cross val score of 0.996.

Training confusion matrix:

array([[1., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0.],
       [0., 0., 1., 0., 0.],
       [0., 0., 0., 1., 0.],
       [0., 0., 0., 0., 1.]])

Testing confusion matrix:

array([[0.997, 0.003 , 0.    , 0., 0.],
       [0.   , 1.    , 0.    , 0., 0.],
       [0.   , 0.    , 1.    , 0., 0.],
       [0.   , 0.    , 0.    , 1., 0.],
       [0.01 , 0.    , 0.007, 0., 0.980]])
$\endgroup$
1
  • $\begingroup$ How imbalanced? If you have $99.5\%$ of cases in one class, then an accuracy score of $99.4\%$ is quite poor. // Are you sure you want to make a hard classification instead of evaluating the probability outputs as statisticians typically prefer to do? How do your models compare on AUC? Better yet, how do they compare on log loss or Brier score? $\endgroup$
    – Dave
    Commented May 26, 2022 at 12:05

3 Answers 3

1
$\begingroup$

You're right to take a critical view of the test score. However, if the test set was correctly chosen, you are not overfitting.

I ran a cross val score on the features and targets before train test split

If you ran the cross validation and any feature extraction on ALL the data before train-test splitting, then what you may have done is accidentally incorporated information from the test set into your model. I've made that mistake before.

You can test your model by running the train test split before anything else, and use a specified random state so you can make it reproducible, and try your model again.

$\endgroup$
2
  • $\begingroup$ I'm a little confused with cross validation. I separated by features and target to X, y. Then I ran a train test split with a random state and stratify=y (my target) since my dataset is imbalanced. Then I ran my cross val score on my model, X, and y. That's how I got the 0.996 cross val score. Is this what you meant by not including the test set? $\endgroup$ Commented Mar 5, 2022 at 7:13
  • 1
    $\begingroup$ could you post your code? That will make it easier to see exactly what is happening, and diagnose. $\endgroup$ Commented Mar 6, 2022 at 8:26
0
$\begingroup$

One simple way to check if your model is overfitting is by plotting the training/validation losses/accuracies along epochs.

If your training loss curve does not decrease or your acc curve does not increase, than your model are not able to fit the training data. You can try to incrase the model capacity, change optimization algorithm, etc.

If your training loss curve decrease or acc curve increase, but the validation ones start to go the different direction. Then your model may overfit to the trianing data. You can mitigate the overfitting by adding regularization, data augmentation, etc.

Finally, when the training and validation losses both go down, it is good.

One caveat is the when you observe your model have low validation accuracy, it is not always "overfitting". You should always investigate training loss first.

$\endgroup$
0
$\begingroup$

A quick check is to ask if you are able to outperform a baseline model that predicts the majority category every time. You mention imbalance. Depending on how imbalanced your data are, you might be able to get better than $99.4\%$ accuracy by just predicting the majority category every time, such as if one category accounts for $99.9\%$ of the cases and you predict that category every time. This would scream out overfitting, as moving from a simple model (predict one category every time) to a complex model (whatever you do) results in a rise in the in-sample performance yet a drop in out-of-sample performance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.