# doubt regarding regularization in linear regression

I was following https://www.analyticsvidhya.com/blog/2015/02/avoid-over-fitting-regularization/ to figure out the basic understanding of regularization in machine learning applications.

Under section "Regularization basics", the authors have commented that a zero of parameter lambda corresponds to over-fit while a value of infinity corresponds to "single mean estimation" (see the attached image for the excerpt ). How is it actually estimating the single mean? Any help. Am I correct when I say that lambda will be a vector rather than a scalar with size same as the number of features taken for the problem at hand ?

The bias term is not included in $||f||_H$. Therefore, it is the only entry that is not penalized by $\lambda$. As $\lambda\to\infty$, all other terms are penalized so heavily that they are forced to zero, leaving the bias term alone in the model. And of course, a model with the bias term alone fits the overall mean.
(And this is the reason why the bias term is not included in $||f||_H$ - we don't want to bias it away from the observations, since the overall mean is typically the best estimated parameter. Plus, penalizing it would mean that the regularized estimate would depend on your units of measure, which is not very helpful.)
• In a regression, the bias term indeed corresponds to the intercept column, i.e., a column consisting of ones. Fitting involves estimating a parameter for this regressor. If there are no other regressors (e.g., they have been "regularized away" by a large $\lambda$) and we fit by minimizing squared residuals, then the parameter estimate will be the mean of the observations. SImilarly, in your screenshot, the sum of squares will be minimized exactly when $f_0=f_0(X_i)$ (be definition, the bias term is independent of the $X_i$) is the mean of the $Y_i$. Jun 14, 2017 at 11:28
• The key property is that if we wish to minimize squared errors to a single value, $\sum(y_i-x)^2\to\min$, then this minimum is achieved exactly at the mean, $x=\frac1N\sum y_i$. Jun 14, 2017 at 11:29