2
$\begingroup$

In a typical neural network, which way is the common way to add regularization?

Assuming regression task, regression error loss is Mean-squared-error

Then we can have two choice of regularization on weights:

  1. $\lambda$ * $\sum ||W||^2$
  2. $\lambda$ * $\textbf{average} ||W||^2$

I have seen most people use the first option, just being curious to ask.

Improve this question
$\endgroup$
4
  • $\begingroup$ depending on what you mean by average, the two should be equivalent as they differ by the scalar value of the number of samples. $\endgroup$
    – meh
    Jul 10, 2018 at 21:02
  • $\begingroup$ What are you summing/averaging over? It's not clear from your expressions $\endgroup$
    – user20160
    Jul 10, 2018 at 21:24
  • $\begingroup$ Is the difference between the two, the second $\lambda$ will be $\lambda/n$ of the original? If so, im not sure if it really matters much. $\endgroup$
    – Anonymous Emu
    Jul 10, 2018 at 22:14
  • $\begingroup$ I agree with @AnonymousEmu, it's just a different scale for lambda variable. With average you just reduce value of the lambda implicitly $\endgroup$
    – itdxer
    Jul 17, 2018 at 15:22

1 Answer 1

Reset to default
1
$\begingroup$

Using the average implicitly rescales $\lambda$. This means that choosing the average or the sum isn't really consequential, because whatever the optimal $\lambda$ is on the mean scale has an equivalent choice of $\lambda$ on the sum scale, and vice versa. $$ \begin{align} \lambda \sum_i w_i^2 &= \lambda\sum_iw_i^2 \\ &= {n\lambda} \left[\frac{1}{n}\sum_iw_i^2 \right]\\ \end{align} $$

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.