4
$\begingroup$

When introducing L2 regularization on my neural network, there is a point during training where the error starts to increase after having reached a value very close to 0. This is due to the fact that when $\Delta_{w}$ gets closer to 0, the most influenced term in weight update become $\lambda w$, that makes the weight go closer to 0, increasing the error. No one seems to point this out when talking about regularization, so I'm a bit confused. What am I missing?

PS: I think that early stopping could be a solution, but is it the right one? And what would you do when there is no validation set to detect when the error stops decreasing and starts increasing?

Improve this question
$\endgroup$
3
  • $\begingroup$ What exactly do you mean by lambda*w? If the weights go to zero, also the L2 penalty term becomes smaller. $\endgroup$
    – Jan Kukacka
    Commented Jan 31, 2018 at 18:06
  • $\begingroup$ You want to trade some error on the training set for a lower error on validation set. You don't want to overfit on the training set. I suggest to do a bit reading on how training, validation and test errors are related and controlled. $\endgroup$
    – Vladislavs Dovgalecs
    Commented Jan 31, 2018 at 18:06
  • $\begingroup$ @JanKukacka No the weight, but the error go to zero, so the penalty term become the most relevant in the formula. Anyway the answer of hxd1011 was very simple and clear, but since usually no one mentions it, i was a bit confused. $\endgroup$
    – Prasqui
    Commented Feb 1, 2018 at 22:03

2 Answers 2

Reset to default
12
$\begingroup$

Adding any regularization (including L2) will increase the error on training set. This is exactly the point of the regularization, where we increase bias and reduce the variance of the model. Hopefully, if we regularized well, as a result, the testing error will be reduced with the regularization.

Here are some related topics.

What problem do shrinkage methods solve?

How to know if a learning curve from SVM model suffers from bias or variance?

Improve this answer
$\endgroup$
3
  • $\begingroup$ Ohhhh very thanks, i totally missed this point. Edit: so adding early stop based on error could be stupid, right? Shall we base it on accuracy? $\endgroup$
    – Prasqui
    Commented Jan 31, 2018 at 17:53
  • $\begingroup$ @Prasqui early stopping is based on the validation error, not the training error! $\endgroup$
    – Jan Kukacka
    Commented Jan 31, 2018 at 17:54
  • $\begingroup$ @JanKukacka Oh, yeah, i know, but i took for granted that both the errors increased, i just wrote more fast than my mind, sorry for stupid point! $\endgroup$
    – Prasqui
    Commented Jan 31, 2018 at 17:57
6
$\begingroup$

Even though @hxd1011 already explained some important things in his answer, I would like answer the second part: What if there is no validation set?

There are methods trying to find the optimal stopping point based solely on the training error. One example is the Optimal Approximation Algorithm by Yinyin Liu and Janusz Starzyk. For details see the article Optimized Approximation Algorithm in Neural Networks Without Overfitting (http://ieeexplore.ieee.org/abstract/document/4471901/).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.