1
$\begingroup$

I was following https://www.analyticsvidhya.com/blog/2015/02/avoid-over-fitting-regularization/ to figure out the basic understanding of regularization in machine learning applications.

Under section "Regularization basics", the authors have commented that a zero of parameter lambda corresponds to over-fit while a value of infinity corresponds to "single mean estimation" (see the attached image for the excerpt ). How is it actually estimating the single mean? Any help.

enter image description here

Am I correct when I say that lambda will be a vector rather than a scalar with size same as the number of features taken for the problem at hand ?

Improve this question
$\endgroup$
1
$\begingroup$

The key point in your screenshot is this:

(except the bias term)

The bias term is not included in $||f||_H$. Therefore, it is the only entry that is not penalized by $\lambda$. As $\lambda\to\infty$, all other terms are penalized so heavily that they are forced to zero, leaving the bias term alone in the model. And of course, a model with the bias term alone fits the overall mean.

(And this is the reason why the bias term is not included in $||f||_H$ - we don't want to bias it away from the observations, since the overall mean is typically the best estimated parameter. Plus, penalizing it would mean that the regularized estimate would depend on your units of measure, which is not very helpful.)

I recommend Statistical Learning with Sparsity by Hastie, Tibshirani and Wainwright.

Improve this answer
$\endgroup$
3
  • $\begingroup$ "a model with the bias term alone fits the overall mean." Could you please explain this? I am asking this because more often than not, the bias term is set to one or some scalar value. How will this fit the overall mean? I mean, we have the data the mean of which could be anything (we may not have computed it yet) then what allows us to assert that this value of bias will be able to model the overall mean $\endgroup$
    – Upendra01
    Jun 14 '17 at 11:23
  • $\begingroup$ In a regression, the bias term indeed corresponds to the intercept column, i.e., a column consisting of ones. Fitting involves estimating a parameter for this regressor. If there are no other regressors (e.g., they have been "regularized away" by a large $\lambda$) and we fit by minimizing squared residuals, then the parameter estimate will be the mean of the observations. SImilarly, in your screenshot, the sum of squares will be minimized exactly when $f_0=f_0(X_i)$ (be definition, the bias term is independent of the $X_i$) is the mean of the $Y_i$. $\endgroup$
    – Stephan Kolassa
    Jun 14 '17 at 11:28
  • $\begingroup$ The key property is that if we wish to minimize squared errors to a single value, $\sum(y_i-x)^2\to\min$, then this minimum is achieved exactly at the mean, $x=\frac1N\sum y_i$. $\endgroup$
    – Stephan Kolassa
    Jun 14 '17 at 11:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.