0
$\begingroup$

I am reading about L2 Regularization. As far as I know we add a thing to the loss function that: $$J(w) = LOSS + \lambda w^T w$$

In the book Deep Learning by Goodfellow et al., they stated "minimizing J(w) results in a choice of weights that make a tradeoff between fitting the training data and being small".

$w^T w$. How is this related to "being small"? Why the weights now tend towards zero rather than any other values?

Improve this question
$\endgroup$
2
  • $\begingroup$ Many duplicates in this list. $\endgroup$
    – kjetil b halvorsen ♦
    Oct 28, 2020 at 12:59
  • 1
    $\begingroup$ I just have a quick look at the list. Most of the post it seems like it explains regularization, but what I want to know is how $w^t w$ mathematically does with regularization. $\endgroup$
    – Minh-Long Luu
    Oct 28, 2020 at 13:09

2 Answers 2

Reset to default
2
$\begingroup$

$w^t w$ term can be written as:

$$ w^t w = \sum_j w_{j}^2 $$

This becomes smaller when $w_j$ is closer to zero. That is why the weight tend to be close to zero due to the regularization term.

Improve this answer
$\endgroup$
1
$\begingroup$

The $w^Tw$ is a quadratic function with minimum at $0$. If the $w$ is "big" (in term of its distance from origin), the $\lambda w^Tw$ term grows and make loss bigger.As we are trying to minimize the loss, we will prefer $w$ to be small (i.e. closer to origin).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.