2
$\begingroup$

In a typical neural network, which way is the common way to add regularization?

Assuming regression task, regression error loss is Mean-squared-error

Then we can have two choice of regularization on weights:

  1. $\lambda$ * $\sum ||W||^2$
  2. $\lambda$ * $\textbf{average} ||W||^2$

I have seen most people use the first option, just being curious to ask.

$\endgroup$
4
  • $\begingroup$ depending on what you mean by average, the two should be equivalent as they differ by the scalar value of the number of samples. $\endgroup$
    – meh
    Jul 10, 2018 at 21:02
  • $\begingroup$ What are you summing/averaging over? It's not clear from your expressions $\endgroup$
    – user20160
    Jul 10, 2018 at 21:24
  • $\begingroup$ Is the difference between the two, the second $\lambda$ will be $\lambda/n$ of the original? If so, im not sure if it really matters much. $\endgroup$ Jul 10, 2018 at 22:14
  • $\begingroup$ I agree with @AnonymousEmu, it's just a different scale for lambda variable. With average you just reduce value of the lambda implicitly $\endgroup$
    – itdxer
    Jul 17, 2018 at 15:22

1 Answer 1

1
$\begingroup$

Using the average implicitly rescales $\lambda$. This means that choosing the average or the sum isn't really consequential, because whatever the optimal $\lambda$ is on the mean scale has an equivalent choice of $\lambda$ on the sum scale, and vice versa. $$ \begin{align} \lambda \sum_i w_i^2 &= \lambda\sum_iw_i^2 \\ &= {n\lambda} \left[\frac{1}{n}\sum_iw_i^2 \right]\\ \end{align} $$

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.