When introducing L2 regularization on my neural network, there is a point during training where the error starts to increase after having reached a value very close to 0. This is due to the fact that when $\Delta_{w}$ gets closer to 0, the most influenced term in weight update become $\lambda w$, that makes the weight go closer to 0, increasing the error. No one seems to point this out when talking about regularization, so I'm a bit confused. What am I missing?

PS: I think that early stopping could be a solution, but is it the right one? And what would you do when there is no validation set to detect when the error stops decreasing and starts increasing?

  • What exactly do you mean by lambda*w? If the weights go to zero, also the L2 penalty term becomes smaller. – Jan Kukacka Jan 31 at 18:06
  • You want to trade some error on the training set for a lower error on validation set. You don't want to overfit on the training set. I suggest to do a bit reading on how training, validation and test errors are related and controlled. – Vladislavs Dovgalecs Jan 31 at 18:06
  • @JanKukacka No the weight, but the error go to zero, so the penalty term become the most relevant in the formula. Anyway the answer of hxd1011 was very simple and clear, but since usually no one mentions it, i was a bit confused. – Prasqui Feb 1 at 22:03
up vote 11 down vote accepted

Adding any regularization (including L2) will increase the error on training set. This is exactly the point of the regularization, where we increase bias and reduce the variance of the model. Hopefully, if we regularized well, as a result, the testing error will be reduced with the regularization.

Here are some related topics.

What problem do shrinkage methods solve?

How to know if a learning curve from SVM model suffers from bias or variance?

  • Ohhhh very thanks, i totally missed this point. Edit: so adding early stop based on error could be stupid, right? Shall we base it on accuracy? – Prasqui Jan 31 at 17:53
  • @Prasqui early stopping is based on the validation error, not the training error! – Jan Kukacka Jan 31 at 17:54
  • @JanKukacka Oh, yeah, i know, but i took for granted that both the errors increased, i just wrote more fast than my mind, sorry for stupid point! – Prasqui Jan 31 at 17:57

Even though @hxd1011 already explained some important things in his answer, I would like answer the second part: What if there is no validation set?

There are methods trying to find the optimal stopping point based solely on the training error. One example is the Optimal Approximation Algorithm by Yinyin Liu and Janusz Starzyk. For details see the article Optimized Approximation Algorithm in Neural Networks Without Overfitting (http://ieeexplore.ieee.org/abstract/document/4471901/).

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.