Suppose that we have a network which performs poorly on the training set, something like ~40% accuracy.

Does it make any sense at all to add a regularisation term in such case? Isn't it only needed when overfitting occurs?

  • $\begingroup$ I think you need to describe the problem in more detail. $\endgroup$ – Michael R. Chernick Apr 8 '17 at 18:47
  • $\begingroup$ If you can only reach 40% accuracy on your training set, it seems like something is wrong with either your model or the optimization procedure, hence overfitting won't be the first of your concerns. $\endgroup$ – galoosh33 Apr 8 '17 at 19:00
  • $\begingroup$ @galoosh33 it's not a problem actually, the model is pretty simple for this problem that's why the low accuracy. The question is more theoretical. $\endgroup$ – Tnp Moe Apr 8 '17 at 20:28
  • $\begingroup$ @MichaelChernick The question is more theoretical, that is, I have a simple model that performs ~40% on training set. The thing is that by adding more and more regularisation there is actually no gains on the test set. The accuracy is always less on all sets (train/val/test). $\endgroup$ – Tnp Moe Apr 8 '17 at 20:30

You can't tell from your model's performance on the training data whether you have overfit. Even your "poor" accuracy of 40% may be due to overfitting and may not generalize. You can't tell a priori what is poor and what isn't. So you might indeed be able to improve your out-of-sample accuracy through regularization.

I'd suggest you partition your data. Fit your model on 75%. Record your in-sample accuracy on these 75%, as well as the out-of-sample accuracy on the remaining 25%. Do this multiple times with different randomly sampled 75%. Your out-of-sample accuracy will usually be lower than in-sample, but if it is often "much" lower, that might be an indicator of overfitting. ("much" is not easily quantified.) In such a case, regularization might be helpful.

  • $\begingroup$ I don't see how overfitting on training set will reduce training set accuracy. $\endgroup$ – Tnp Moe Apr 8 '17 at 18:58
  • 1
    $\begingroup$ I didn't claim that. Overfitting might reduce out-of-sample accuracy. $\endgroup$ – Stephan Kolassa Apr 8 '17 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.