Very basic question here, but I would like to understand (not mathematically) how the fact to add a "penalty" (sum of squared coeff. times a scalar) to the residual sum of square can reduce big coefficients ? thanks !


1 Answer 1


Because your "penalty" represenation of the minimization problem is just the langrange form of a constraint optimization problem:

Assume centered variables. In both cases, lasso and ridge, your unconstrained target function is then the usual sum of squared residuals; i.e. given $p$ regressors you minimize: $$RSS(\boldsymbol{\beta}) = \sum_{i=1}^n (y_i-(x_{i,1}\beta_1 +\dots +x_{i,p}\beta_p))^2.$$ over all $\boldsymbol{\beta} =(\beta_1,\dots, \beta_p)$.

Now, in the case of a ridge regression you minimize $RSS(\boldsymbol{\beta})$ such that $$\sum_{i=1}^p\beta_p^2 \leq t_{ridge},$$ for some value of $t_{ridge}\geq 0$. For small values of $t_{ridge}$ it will be impossible to derive the same solution as in standard least square scenario in which case you only minimize $RSS(\boldsymbol{\beta})$ -- Think about $t_{ridge}=0$ then the only possible solution can be $\beta_1\equiv \dots \equiv \beta_p = 0$.

On the other hand, in the case of the lasso, you minimize $RSS(\boldsymbol{\beta})$ under the constraint $$\sum_{i=1}^p|\beta_p| \leq t_{lasso},$$ for some value of $t_{lasso}\geq 0$.

Both constrained optimization problems can be equivalently forumlated in terms of an unconstrained optimization problem, i.e. for the lasso: you can equivalently minimize

$$\sum_{i=1}^n (y_i-(x_{i,1}\beta_1 +\dots +x_{i,p}\beta_p))^2 + \lambda_{lasso}\sum_{i=1}^p|\beta_p|.$$

  • $\begingroup$ Thanks, i'll have to deep into the "constrained to unconstrained" part but i got the idea $\endgroup$
    – TmSmth
    Commented Jul 23, 2018 at 9:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.