Reason for not shrinking the bias (intercept) term in regression

Question

For a linear model $y=\beta_0+x\beta+\varepsilon$, the shrinkage term is always $P(\beta) $.

What is the reason that we do not shrink the bias (intercept) term $\beta_0$? Should we shrink the bias term in the neural network models?

Answer 1

The Elements of Statistical Learning by Hastie et al. define ridge regression as follows (Section 3.4.1, equation 3.41): $$\hat \beta{}^\mathrm{ridge} = \underset{\beta}{\mathrm{argmin}}\left\{\sum_{i=1}^N(y_i - \beta_0 - \sum_{j=1}^p x_{ij}\beta_j)^2 + \lambda \sum_{j=1}^p \beta_j^2\right\},$$ i.e. explicitly exclude the intercept term $\beta_0$ from the ridge penalty.

Then they write:

[...] notice that the intercept $\beta_0$ has been left out of the penalty term. Penalization of the intercept would make the procedure depend on the origin chosen for $Y$; that is, adding a constant $c$ to each of the targets $y_i$ would not simply result in a shift of the predictions by the same amount $c$.

Indeed, in the presence of the intercept term, adding $c$ to all $y_i$ will simply lead to $\beta_0$ increasing by $c$ as well and correspondingly all predicted values $\hat y_i$ will also increase by $c$. This is not true if the intercept is penalized: $\beta_0$ will have to increase by less than $c$.

In fact, there are several nice and convenient properties of linear regression that depend on there being a proper (unpenalized) intercept term. E.g. the average value of $y_i$ and the average value of $\hat y_i$ are equal, and (consequently) the squared multiple correlation coefficient $R$ is equal to the coefficient of determination $R^2$: $$(R)^2 = \text{cor}^2(\hat {\mathbf y}, \mathbf y) = \frac{\|\hat{\mathbf y}\|^2}{\|\mathbf y\|^2} = R^2,$$ see e.g. this thread for an explanation: Geometric interpretation of multiple correlation coefficient $R$ and coefficient of determination $R^2$.

Penalizing the intercept would lead to all of that not being true anymore.

Answer 2

Recall the purpose of shrinkage or regularization. It is to prevent the learning algorithm to overfit the training data or equivalently - prevent from picking arbitrarily large parameter values. This is more likely for datasets with more than few training examples in the presence of noise (very interesting discussion about presence of noise and its impact is discussed in "Learning from Data" by Yaser Abu-Mustafa). A model learned on noisy data with no regularization will likely perform poorly on some unseen data points.

With this in mind, imagine you have 2D data points which you want to classify in two classes. Having all but the bias parameters fixed, varying the bias term will just move the boundary up or down. You can generalize this to a higher dimensional space.

The learning algorithm cannot put arbitrarily large values for the bias term since this will result in possibly gross loss value (the model will not fit the training data). In other words, given some training set, you (or a learning algorithm) cannot move the plane arbitrarily far away from the true one.

So, there is no reason to shrink the bias term, the learning algorithm will find the good one without a risk of overfitting.

A final note: I saw in some paper that when working in high-dimensional spaces for classification, there is no strict need to model the bias term. This might work for linearly separable data since with more dimensions added, there are more possibilities to separate the two classes.

Answer 3

The intercept term is absolutely not immune to shrinkage. The general "shrinkage" (i.e. regularization) formulation puts the regularization term in the loss function, e.g.:

$RSS(\beta) = \|y_i - X_i \beta \|^2$

$RegularizedLoss(\beta) = RSS(\beta) - \lambda f(\beta)$

Where $f(\beta)$ is usually related to a lebesgue norm, and $\lambda$ is a scalar that controls how much weight we put on the shrinkage term.

By putting the shrinkage term in the loss function like this, it has an effect on all the coefficients in the model. I suspect that your question arises from a confusion about notation in which the $\beta$ (in $P(\beta)$) is a vector of all the coefficients, inclusive of $\beta_0$. Your linear model would probably be better written as $y = X \beta + \epsilon$ where $X$ is the "design matrix," by which I mean it is your data with a column of $1's$ appended to the left hand side (to take the intercept).

Now, I can't speak to regularization for neural networks. It's possible that for neural networks you want to avoid shrinkage of the bias term or otherwise design the regularized loss function differently from the formulation I described above. I just don't know. But I strongly suspect that the weights and bias terms are regularized together.

Answer 4

Suppose one of the predictors $x_i$ happens to have the same nonzero value across all training examples. We would like its coefficient $\beta_i$ to be estimated as zero. To see why, suppose $\beta_i$ is not zero, and $x_i$ takes a different value in some test example. Our prediction will change by some arbitrary amount that has no justification in the training data.

By not shrinking the intercept $\beta_0$ in ridge regression, we ensure that $\beta_i$ will be zero. If we did shrink the intercept, then $\beta_i$ will not be zero, since $x_i$ plays the role of a second intercept and will split up $\beta_0$.

You might argue that we should normalize $x_i$ by subtracting its mean and making it zero across all training examples. However, that act of normalization implicitly assumes the intercept is excluded from the ridge penalty.

Answer 5

I'm not sure the above answer by David Marx is quite right; according to Andrew Ng, by convention the bias/intercept coefficient is typically not regularized in a linear regression, and in any case whether it is regularized or not does not make a significant difference.

Answer 6

I'll give the simplest explanation, then expand.

Suppose you shrink to zero, then your model effectively becomes: $$y_t=\varepsilon_t$$ Just one problem with this model: $E[\varepsilon_t]=E[y_t]\ne 0$, which violates exogeneity assumption of the linear regression. Hence, the estimated coefficients will not have nice properties, such as unbiasedness.

This demonstrates the main purpose of the intercept: to capture the mean. I think that a lot of people do not realize the importance of the intercept in the linear regression. It's often looked down as a less sexy little brother of the "real" $\beta$ of the predictor. However, as you may know from "regression through the origin" dropping the intercept from the model often leads to undesirable consequences.

Now, for the completeness if you shrink all bona fide coefficients $\beta$ and keep the intercept $\beta_0$ out, you get this: $$y_t=\beta_0+\varepsilon_t$$ $$E[y_t]=\beta_0+E[\varepsilon_t]$$ Here, we still have $E[\varepsilon_t]=0$ because the intercept will capture the mean of the data $\beta_0=\mu=E[y_t]$.

This model is not as sexy as the original model, it's rather silly, in fact. However, it is a legit model. You could run ANOVA on it, for example.

Concluding, you need to keep the intercept out of shrinkage so that it does what it is intended for: capture the mean of the series $\beta_0=E[y_t]$