Not clear why adding additional features (not just transformations) reduces model bias in statistical machine learning

Question

Setup

In a regression setting, consider the issue of model bias (rather than sampling bias or other statistical biases). If the true data generating process looks like $Y = f(X) + \mathrm{noise}$ and our goal is to estimate $f$ well, we might fit a particular statistical model or machine learning algorithm to the available dataset and get an estimate $\hat f$. For some procedures, perhaps at many points $X=x_0$ it turns out that $\mathbb{E}(\hat f(x_0)) \approx f(x_0)$, and we say the model bias is low. In other words: on average across many repetitions of taking a sample and fitting the model, the fitted models tend to match the true regression function across most of the range of $X$.

For other procedures, perhaps it turns out that $\mathbb{E}(\hat f(x_0)) \neq f(x_0)$ at many points $X=x_0$, and we say the model bias is high(er). This concept is often illustrated with the example of a nonlinear $f$ and a single feature $X$. For instance, if the true function is $f(X) = X^2$ on the range $X \in (-1, 1)$, but we try to fit $\hat f$ with a simple linear regression, there's no way a straight line can match the true $f$ for most of the relevant range of $X$, not even in expectation. So a simple linear model would have relatively high model bias for the true regression function here, compared to (for instance) polynomial regression or spline models which could have low or no model bias.

Question

It is clear to me how adding polynomials (or other nonlinear transformations) of the features could reduce model bias. So far so good. But there is also the claim that adding more features (entirely new measurements, not transformations of features already in the model) can also reduce model bias. I don't see why. In my humble opinion it will not fix it since linear regression will stay linear if a new feature is added. With one feature we will have a straight line in a plane. By adding a totally new feature, we will still have a "straight" fitted function (a plane with no nonlinearities) in a 3 dimensional space.

So in what sense is it possible for model bias to be reduced when we add new features, not just nonlinear transformations on existing features?

Answer 1

You wrote:

With one feature we will have a straight line in a plain, and by adding one feature we will have a straight plane in a 3 dimensional space.

Imagine that your response variable, $Y$, truly depends on two features, $X_1$ and $X_2$. Let's even say that a linear model is true:

$Y = f(X_1, X_2) + \mathrm{noise} = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \mathrm{noise}$

If these betas are nonzero, then in the 3D space, $f(X_1, X_2)$ is a tilted plane. It's not curved (no polynomials, interaction terms, etc.). But it sits at an angle to each of the $X$ axes.

Now, let's say that you start by ignoring $X_2$ and just trying to fit the model

$Y = \beta_0 + \beta_1 X_1 + \mathrm{noise}$

Then you will also get a plane as you pointed out. This plane is tilted with respect to the $X_1$ axis, but not with respect to the $X_2$ axis.

This is a biased model. Since you don't include $X_2$ in your model, there's no way for the fitted plane $Y = \hat\beta_0 + \hat\beta_1 X_1$ to match the true plane $Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2$, not even on average across different samples from the population.

Answer 2

If you think about it, adding polynomial features is somewhat equivalent to polynomial regression. Instead of adding a polynomial term to the model, you add it to the features. You will end up with the same equations. A polynomial model has less bias because it can fit more complex functions.

Linear regression with two components: $$ LinReg(x_1, x_2) = a_0 + a_1 x_1 + a_2 x_2$$

Polynomial regressions with one component: $$ PolyReg(x_1) = a_0 + a_1 x_1 + a_2 x_1^2$$

If you set $x_2 = x_1^2$ you end up with the same expression.