6
$\begingroup$

Training a simple neural network over a very sparse matrix (Has 2400 features and 18000 train rows) for a binary classification problem. At the end of 1st epoch validation loss started to increase, whereas validation accuracy is also increasing. Can i call this over fitting? I'm thinking of stopping the training after 6th epoch. My criteria would be: stop if the accuracy is decreasing. Is there something really wrong going on?

ps: I have perfectly balanced binary classification dataset and a random classifier would result around %50 accuracy.

Getting this figure

$\endgroup$
  • $\begingroup$ Based on the second graph you would say that your model is overfitting, since a similar increase in accuracy from the training set does not happen on the test set. $\endgroup$ – user2974951 Sep 24 '18 at 12:31
  • $\begingroup$ Yes but the accuracy is increasing a bit, even to the %65 accuracy level.. If i stop my network when validation loss is increasing i get %55 accuracy. Wouldn't it be better to have %65 accurate network over %55? I am questioning the stopping criteria. $\endgroup$ – betelgeuse Sep 24 '18 at 12:39
  • $\begingroup$ I would say from epoch 3 forward the increase is marginal. Maybe this validation loss criterion is not the best in your case. $\endgroup$ – user2974951 Sep 24 '18 at 12:43
4
$\begingroup$

Yes, absolutely. First of all, overfitting is best judged by looking at loss, rather than accuracy, for a series of reasons including the fact that accuracy is not a good way to estimate the performance of classification models. See here:

https://stats.stackexchange.com/a/312787/58675

Why is accuracy not the best measure for assessing classification models?

Classification probability threshold

Secondly, even if you use accuracy, rather than loss, to judge overfitting (and you shouldn't), you can't just look at the (smoothed) derivative of accuracy on the test curve, i.e., if it's increasing on average or not. You should first of all look at the gap between training accuracy and test accuracy. And in your case this gap is very large: you'd better use a scale which starts either at 0, or at the accuracy of the random classifier (i.e., the classifiers which assigns each instance to the majority class), but even with your scale, we're talking a training accuracy of nearly 100%, vs. a test accuracy which doesn't even get to 65%.

TL;DR: you don't want to hear it, but your model is as overfit as they get.

PS: you're focusing on the wrong problem. The issue here is not whether to do early stopping at the 1th epoch for a test accuracy of 55%, or whether to stop at epoch 7 for an accuracy of 65%. The real issue here is that your training accuracy (but again, I would focus on the test loss) is way too high with respect to your test accuracy. 55%, 65% or even 75% are all crap with respect to 99%. This is a textbook case of overfitting. You need to do something about it, not focus on the "less worse" epoch for early stopping.

$\endgroup$
  • $\begingroup$ What would be the best way to approach to the problem i have tried noises, dropouts etc? I wouldn't expect a better test accuracy then %65 for this dataset. Should i be preventing the network to get a high train accuracy? Or should i prevent the gap to be widen a lot? $\endgroup$ – betelgeuse Sep 24 '18 at 12:56
  • $\begingroup$ as a side note: I have perfectly balanced binary classification dataset. $\endgroup$ – betelgeuse Sep 24 '18 at 13:04
  • $\begingroup$ it's impossible to answer if you don't a) describe in detail what's your problem, b) explain the why you don't expect a better than 65% test accuracy for this dataset. Of course "I have a hunch that test accuracy can't better than 65%" is not a valid reason, while "I have extensive bibliographic evidence that SOTA on this dataset, or on a extremely similar one, cannot exceed 65%" is a valid reason. See stats.stackexchange.com/a/363895/58675 for some references on SOTA results for different ML problems (alas, mostly NLP and CV). And btw, note that how to reduce the test error 1/ $\endgroup$ – DeltaIV Sep 24 '18 at 16:18
  • $\begingroup$ 2/ on your problem is a completely different question from what you asked. You need to ask a new question and describe your actual dataset & model much more in detail, as well as describe extensively all the things you have already tried. For your own sanity, I hope you're not using Jupyter notebooks to keep track of this kind of extensive numerical experiments. $\endgroup$ – DeltaIV Sep 24 '18 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.