3
$\begingroup$

If I set the regularization parameter to 0, does it essentially mean I'm not applying regularization (I've boxed the regularization bits in red)?

Also, what is this type of regularization called?

enter image description here

$\endgroup$
5
$\begingroup$
  1. When $\lambda=0$, no regularization is applied. Zero multiplied by any number is zero.
  2. Penalizing squared coefficients is sometimes called ridge regression, or L2 regularization. Other kinds of regularization include the LASSO or L1 regularization, which penalizes the absolute value of the coefficients. Elastic net regression is a compromise between the two, which includes both LASSO and includes both ridge regressions as special cases.
$\endgroup$
  • $\begingroup$ Thanks for your detailed answer! Ahh, so by applying this type of regularization to a linear regression problem, can one say it's a ridge regression problem? Thanks for explaining a bit of the other types! $\endgroup$ – KingPolygon Jul 17 '15 at 21:20
2
$\begingroup$

Yes. The regularization term vanishes when $\lambda = 0$.

$\endgroup$
1
$\begingroup$

Exactly! You can think of the cost function as having two parts:

  • A "cost" associated with the difference between your model and the data, plus
  • A "cost" for having two many non-zero coefficients.

The $\lambda$ term sets the weight on the second part of the cost function. If it's zero, then that entire term goes to zero $\forall x, 0x=0$) and it is as if regularization were never applied.

$\endgroup$
0
$\begingroup$

Consider an extreme case in which 2 of your variables are absolutely collinear, without regularization their coefficient estimates will explode into infinite solution space, but with regularization, even with the regularization parameter tending to zero, we will have stable values. One of the uses of regularization is to tackle multicollinearity.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.