3
$\begingroup$

I'd like to preface my question by saying that I understand that the reason for introducing a regularization term to the training criterion is to avoid overfitting, with the effect that (usually) model performance on the test set is improved, i.e. the goal being to increase the model's 'generalization' capacity.

However, while exploring different hyperparameters in my neural network models, I sometimes get the impression that my network's performance on the training set can also benefit from adding regularization -- L2 regularization, in my case -- compared to not adding any penalty at all.

I realize that the most likely explanation is that I just see patterns where there are none, but I wanted to know if there is any formal explanation why, hypothetically, a regularization penalty could help during the training phase as well -- or, contrary, why it is impossible for such a thing to happen.

Note that by 'help during training' I don't necessarily mean 'final performance on the train data', but perhaps just 'faster convergence' to a good solution.

So for example (I'm completely guessing here, my apologies), can we exclude the possibility that adding a penalty, which 'pulls' the weights towards a prior value, might help during training by allowing the model to escape a local minimum or saddle point slightly faster?

As I mentioned already: I can see why a regularization penalty shouldn't (or rather: cannot) help with training set performance, but that understanding is based on my limited 'static' understanding of the model.

Once I include the training process I fail to see if there would be any way that regularization could be useful in achieving faster convergence/less likelihood of training getting 'stuck'.

$\endgroup$
4
  • 1
    $\begingroup$ "So for example (I'm completely guessing here, my apologies), can we exclude the possibility that adding a penalty, which 'pulls' the weights towards a prior value, might help during training by allowing the model to escape a local minimum or saddle point slightly faster?" This is probably what's happening: the penalty changes the geometry of the response surface. $\endgroup$
    – Sycorax
    Feb 13, 2017 at 18:59
  • $\begingroup$ @Sycorax But that seems to be contrary to the quite clear statement that I've seen before: that regularization only has a (positive) effect on validation performance, and can only /decrease/ training performance. $\endgroup$ Feb 13, 2017 at 19:10
  • $\begingroup$ You're the one with experimental evidence to the contrary, so either the claim you make in your comment clearly can't be universally true or you've made a mistake in your experiment. But both are not true. $\endgroup$
    – Sycorax
    Feb 13, 2017 at 20:22
  • $\begingroup$ Fair enough. I guess I was hoping for an ex cathedra answer along the lines of "that is impossible" or "that is provably true", but I'll also settle for empirical evidence. Thanks for the feedback! $\endgroup$ Feb 13, 2017 at 22:07

1 Answer 1

0
$\begingroup$

It is impossible, except for when it is possible.

It is impossible because minimizing the loss function literally finds the parameter values that are the best. If regularized parameter values were better (lower loss), then the minimization of the unpenalized loss function would have found those parameter values.

However...

...neural networks have nasty loss functions where, unlike the closed-form solution for OLS regression, we have to look around the loss function until we find a place that is adequate so that we can declare, "This is the model," perhaps a global minimum but perhaps a local minimum. If you only get a local (but not global) minimum with the unregularized model, I find it completely plausible that the regularized model could wind up optimized by parameters that result in lower loss than the ones from the local minimum of the unregularized model, as you described.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.