# L2 regularization and its intuition

I am reading about L2 Regularization. As far as I know we add a thing to the loss function that: $$J(w) = LOSS + \lambda w^T w$$

In the book Deep Learning by Goodfellow et al., they stated "minimizing J(w) results in a choice of weights that make a tradeoff between fitting the training data and being small".

$$w^T w$$. How is this related to "being small"? Why the weights now tend towards zero rather than any other values?

• Oct 28, 2020 at 12:59
• I just have a quick look at the list. Most of the post it seems like it explains regularization, but what I want to know is how $w^t w$ mathematically does with regularization. Oct 28, 2020 at 13:09

$$w^t w$$ term can be written as:
$$w^t w = \sum_j w_{j}^2$$
This becomes smaller when $$w_j$$ is closer to zero. That is why the weight tend to be close to zero due to the regularization term.
The $$w^Tw$$ is a quadratic function with minimum at $$0$$. If the $$w$$ is "big" (in term of its distance from origin), the $$\lambda w^Tw$$ term grows and make loss bigger.As we are trying to minimize the loss, we will prefer $$w$$ to be small (i.e. closer to origin).