15
$\begingroup$

When implementing a neural net (or other learning algorithm) often we want to regularize our parameters $\theta_i$ via L2 regularization. We do this usually by adding a regularization term to the cost function like so: $$\text{cost}=\frac{1}{m}\sum_{i=0}^m\text{loss}_m +\frac{\lambda}{2m}\sum_{i=1}^n (\theta_i)^2$$

We then proceed to minimize this cost function and hopefully when we've reached a minimum, we get a model which has less overfitting than a model without regularization. As far as I know, this is the L2 regularization method (and the one implemented in deep learning libraries). Let me know if I have made any errors.

My question is this: since the regularization factor has nothing accounting for the total number of parameters in the model, it seems to me that with more parameters, the larger that second term will naturally be. If a model has 300 million parameters, for example, and I set $\lambda=1$, that second term might be huge. Is it standard practice, then, to reduce $\lambda$ in some way to account for the massive number of parameters in the model, or is it ok to simply accept starting off with a huge cost? It seems to me that if we do not somehow scale $\lambda$ inversely with the number of parameters, that using a huge number of parameters, while keeping $\lambda$ constant, means that we will have a much stronger regularization effect since the second term will force the parameters $\theta_i \approx 0$ with much more rigor. That second term will dominate over the first term. I have not seen any mention of doing this in any of the resources that I've come across though, so I wonder if my analysis is fundamentally wrong somewhere.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Cross validation (hmm, there's a Q&A website sounding something like that) is often used to choose an optimal, or at least good, value of $\lambda$, or $\lambda/(2m)$. $\endgroup$
    – Mark L. Stone
    Commented Dec 3, 2017 at 22:30
  • 1
    $\begingroup$ Yes, you can use cross validation to find an optimal $\lambda$, but consider if a neural network takes a week to train, there are so many possible values of lambda to try (infinitely many actually) it would be good if there's some rules of thumb. If we try $\lambda=1$ as a rule of thumb, we might be way off if we go from a network with 20k parameters to one with 20m parameters if my analysis is correct. Thanks for the sarcasm though, it's very helpful. $\endgroup$
    – enumaris
    Commented Dec 3, 2017 at 22:37
  • $\begingroup$ I've been thinking about this question. Any more pointers? Anyone know how it's implemented in tf etc? $\endgroup$
    – tea_pea
    Commented Jul 29, 2018 at 14:42
  • $\begingroup$ Ok, looked through the the tensorflow and couldn't find anything that divided by number of weights. So I suppose it's not the way to go. $\endgroup$
    – tea_pea
    Commented Jul 29, 2018 at 15:10

2 Answers 2

Reset to default
8
$\begingroup$

You are absolutely right in your observation that the number of parameters will affect the regularization cost.

I don't think there are any rule-of-thumb values for $\lambda$ (but $\lambda=1$ would be considered large). If cross-validation is too time-consuming, you could hold-out a part of the training data and tune $\lambda$ using early stopping. You would still need to try several values for $\lambda$ common practice is to try something like $0.01, 0.02,\ldots,0.4$.

For really large networks, it might be more convenient to use other regularization methods, like dropout, instead of $\ell_2$ regularization.

Improve this answer
$\endgroup$
5
  • 2
    $\begingroup$ I asked this question because I have a network with 35 million parameters, and when I use $\lambda=0.01$ the cost started around 50 for binary classification (which for random guessing would be cost = 0.69 without regularization). I wondered if that meant that all the optimizing would be simply crushing all the parameters to 0 rather than actually learning anything or if it's generally no big deal to start with a big cost like that. Wondering if there's some general wisdom in applying L2 regularization that applied to this situation. $\endgroup$
    – enumaris
    Commented Dec 4, 2017 at 1:14
  • $\begingroup$ you'd have to make $\lambda$ really small $\sim 10^{-4}$. I would look into dropout instead of $\ell_2$ regularization. $\endgroup$
    – A.D
    Commented Dec 4, 2017 at 3:01
  • $\begingroup$ I have in fact implemented dropout, but I find sometimes dropout doesn't give me the results I want so I wanna try both. Thanks for the answer! $\endgroup$
    – enumaris
    Commented Dec 4, 2017 at 19:11
  • $\begingroup$ The number of parameters may affect the regularization cost, but it won't "crush" all the parameters to zero. That is because the derivative of total cost w.r.t. each individual parameter has two components: the derivative w.r.t. the model cost and the derivative w.r.t. the regularization cost, and the latter does NOT depend upon the number of parameters. So, when the derivative for our selected parameter gets set to 0 (in effect) the other parameters only enter through their influence on the model cost, not through their influence on the regularization cost. $\endgroup$
    – jbowman
    Commented Dec 5, 2017 at 0:00
  • $\begingroup$ This is the only place online that I've found an example of an appropriate value to try for $\lambda$. I wasn't even sure the typical order of magnitude to try. If no one has already, I think it would be a valuable contribution to research for someone to publish a paper with the optimal value of $\lambda$ for different problems (e.g. MNIST) and NN architectures, just to see if they vary wildly, or if there is a good heuristic. $\endgroup$
    – Joe
    Commented Mar 27, 2020 at 18:23
0
$\begingroup$

You should not care for the final loss but rather about its derivative. Each parameter's derivative with regards to the loss will see its derivative corrected by the regularisation but this term won't affect the backprop.

So overall the loss will be huge but only the error part (that does not really scale up with the number of parameters) will affect the backprop.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.