6
$\begingroup$

Does bias eventually increase with model complexity?

Bias-variance tradeoff

Reasoning behind the question:

If I understand it correctly, "bias" measures the discrepancy between the expected value of our model's ($\hat{f}$) prediction and the expected value of the true, underlying model $f$. We can start by creating a simple model with high bias. Bias quickly drops as we train our model with the data since $\hat{f}$ starts to resemble the true $f$ function. However, our model may become too complex, too wiggly compared to the true model. Therefore introducing high bias again.

Edit:

I found a clue in An Introduction to Statistical Learning (2nd edition). FIGURE 2.10. on page 33 together with FIGURE 2.12. on page 36 visually suggest that the bias actually doesn't eventually increase with model complexity.

Flexibility vs bias

The plot on the left shows different models fitted to the data and the black line is a true relationship. The plot on the right corresponds to the data on the left.

$\endgroup$
3
  • 1
    $\begingroup$ Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. $\endgroup$ Commented Feb 23, 2022 at 19:17
  • $\begingroup$ @ClosedLimelikeCurves, done. $\endgroup$
    – Glue
    Commented Feb 23, 2022 at 21:17
  • $\begingroup$ "Wiggly" is not the same as bias, but there is one sense in which I can see this: for certain very flexible models, the model as fit seems to avoid the standard bias-variance trade-off even while the model as defined doesn't (see, e.g., arxiv.org/abs/2109.02355). The theory on this is still very cutting-edge, but it seems to be at least in part because some bias comes "back in" as a consequence of the specific fitting algorithms used for these highly complex learners in the over-parameterized (very high complexity vs data) regime. $\endgroup$
    – mweylandt
    Commented Mar 1, 2022 at 0:03

2 Answers 2

2
$\begingroup$

I was wondering the same thing. What you have to realize though, is that the bias is defined for the expectation of the prediction for a given x over all possible training sets (D):

\begin{equation} D=\left\{\left(x_{1}, y_{1}\right) \ldots,\left(x_{n}, y_{n}\right)\right\} \end{equation}

\begin{equation} \operatorname{Bias}_{D}[\hat{f}(x ; D)]=\mathrm{E}_{D}[\hat{f}(x ; D)]-f(x) \end{equation}

This means that the expectation ranges over different choices of the training set. In your example (figure 2.10), a specific training set is given. Indeed, the prediction of the flexible model (green line) is way off the real function. Now imagine that you would create multiple training sets, and create multiple of these green lines. The average of these green lines would almost perfectly allign the black line (real model), even better than the yellow line would. The expectation of the prediction of all possible training sets for the more flexibel model is closer to the real model and the bias is therefore lower.

$\endgroup$
1
  • $\begingroup$ Why do the average of the green lines almost perfectly align with the black line? I suppose it will have to do with the model being equally likely to overpredict and underpredict on non-train data, but it's not clear to me why that is necessarily the case. $\endgroup$
    – user35734
    Commented Jun 10 at 16:02
0
$\begingroup$

It depends, is all I can say.

As a rule of thumb, bias decreases in a model as you add more parameters, but there can be some weird exceptions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.