Can adding features to linear regression increase model bias?

Question

I think it may be possible to increase bias in a linear regression model by adding bad features, but I can't think of an example that makes sense intuitively on the basis of the bias-variance decomposition. For instance, using the formulation of bias variance decomposition here, the square bias term is $(\bar g(x) - f(x))^2$. I can't think of a scenario where this term would increase when adding an extra feature, since I think adding a bad feature would force its associated parameter to be zero for both $\bar g$ and $f$.

Edited

Answer 1

I wrote a paper about this. An early version of it is on arxiv here: https://arxiv.org/abs/2003.08449 (edit: The full published version is now available open access here: https://journals.sagepub.com/doi/full/10.1177/0962280221995963).

The short of it is yes under some conditions. Namely, there has to be some bias already in the model. If you model is already biased then bias can be increase by adding variables.

As an example, suppose we are interested in the effect of some treatment $A$ on the outcome $Y$. For simplicity, say this effect is just the regression coefficient $\beta$. Bias will be with respect to this parameter. The easiest way to increase the bias of the estimate of $\beta$, $\hat{\beta}$ is by adding variables which explain much of the variation in the treatment ($A$) and very little variation in the outcome ($Y$), except through $A$ of course.

The reason that this happens is due to the geometry of least squares. By the FWL theorem, we can always think of estimates from multivariate regression as equivalent to the estimate from a simple regression of:

1. The residuals of the outcome from the regression of $Y$ on all the control variables and

2. The residuals of the treatment from the regression of $A$ on all the control variables

Suppose we have a variable which only has an effect on $Y$ through $A$ (an instrumental variable) and there are no interactions. When we add this variable to the regression, whatever direction the old estimate was relative to the truth gets amplified (see figure 4 in the above paper for a graph of what this looks like).

In general, a variable can cause bias amplification even if it does explain some variance of the outcome not through the treatment. In these cases, including a variable is a trade-off. By including it you remove the bias do to excluding said variable, but you also amplify the bias of the remaining biases. Whether or not there is bias amplification in the end depends on which effect is stronger. In extreme cases, especially those where there are plenty of control variables already, the amplification effect can tend to dominate because the amplification effect is hyperbolic and always in the same direction. The effect of removing omitted variable bias on the other hand is linear in these settings and may go in either direction.

There is a small literature of papers looking at these effects. The original two which investigated pure instruments are

i) Pearl (2011) https://ftp.cs.ucla.edu/pub/stat_ser/r386.pdf
ii) Wooldridge (2006) http://econ.msu.edu/faculty/wooldridge/docs/treat1r6.pdf

But has since been extended to a slightly broader class of models and under more flexible conditions.

To your point about bias-variance trade-off. In most cases, the variance will also increase when you have bias amplification as well unfortunately.