Say I am training a neural network for image classification of cats and dogs, in a dataset of 1000 for example. I train the network on 800 examples, and then I test it with the remaining 200 examples, and I get training and test accuracies of 94% and 73% --> 21% gap

Applying regularization should fix this train-test accuracy gap up to some extent. Would it be right to assume that there is always the possibility of a trade-off between training accuracy and test accuracy, by incrementing the amount of regularization being applied to the network?

In other words, is it correct to assume that there exists an specific setting for my regularization hyperparameters, that would allow to "almost completely" reduce that 21% gap, to something like 1% or 2% gap? At the cost of reducing training accuracy, getting for instance 79%-77% train and test accuracies.

I've found this other simmilar issue validation/training accuracy and overfitting, however I don't find any of the answers accurate enough to the particular question.


The accuracy gap between train and test data is caused by fitting:

  1. noises in train data (hence over-fit)
  2. features exist only in train data

The regularisation can control fitting to noises.

But for the second point, you may need a process like feature selection.

  • $\begingroup$ So @Sixiang.Hu, assuming that point 2. does not exist in this case, i.e. training and test data are rather simmilar, would then be always possible to get rid of that noise/overfitting gap by applying the correct amount of regularization? $\endgroup$
    – sdiabr
    Apr 4 '18 at 19:10
  • $\begingroup$ Get rid of noise fully is not really practically imo. The reason is "get rid of noise" fully means you can "model" noise ( so that it can be regularized ), which is not practical. And this is why I used "control fitting to noises" as oppose to "fully get rid of noise". $\endgroup$
    – Sixiang.Hu
    Apr 4 '18 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.