How Ridge or Lasso regression really work? Very basic question here, but I would like to understand (not mathematically)  how the fact to add a "penalty" (sum of squared coeff. times a scalar) to the residual sum of square can reduce big coefficients ? thanks !
 A: Because your "penalty" represenation of the minimization problem is just the langrange form of a constraint optimization problem:
Assume centered variables.
 In both cases, lasso and ridge, your unconstrained target function is then the usual sum of squared residuals; i.e. given $p$ regressors you minimize: 
$$RSS(\boldsymbol{\beta}) = \sum_{i=1}^n (y_i-(x_{i,1}\beta_1 +\dots +x_{i,p}\beta_p))^2.$$
over all $\boldsymbol{\beta} =(\beta_1,\dots, \beta_p)$. 
Now, in the case of a ridge regression you minimize $RSS(\boldsymbol{\beta})$ such that $$\sum_{i=1}^p\beta_p^2 \leq t_{ridge},$$
for some value of $t_{ridge}\geq 0$. For small values of $t_{ridge}$ it will be impossible to derive the same solution as in standard least square scenario in which case you only minimize $RSS(\boldsymbol{\beta})$ -- Think about $t_{ridge}=0$ then the only possible solution can be $\beta_1\equiv \dots \equiv \beta_p = 0$. 
On the other hand, in the case of the lasso, you minimize $RSS(\boldsymbol{\beta})$  under the constraint 
$$\sum_{i=1}^p|\beta_p| \leq t_{lasso},$$
for some value of $t_{lasso}\geq 0$.
Both constrained optimization problems can be equivalently forumlated in terms of an unconstrained optimization problem, i.e. for the lasso: you can equivalently minimize 
$$\sum_{i=1}^n (y_i-(x_{i,1}\beta_1 +\dots +x_{i,p}\beta_p))^2 + \lambda_{lasso}\sum_{i=1}^p|\beta_p|.$$
