Bias vs reducible error

Question

I encountered a question while learning:

While doing a homework assignment, you fit a Linear Model to your data set. You are thinking about changing the Linear Model to a Quadratic one.

• Using the Quadratic Model will decrease your Irreducible Error
• (Correct answer) Using the Quadratic Model will decrease the Bias of your model
• Using the Quadratic Model will decrease the Variance of your model
• Using the Quadratic Model will decrease your Reducible Error

And an explanation for it:

Introducing the quadratic term will make your model more complicated. More complicated models typically have lower bias at the cost of higher variance. This has an unclear effect on Reducible Error (could go up or down) and no effect on Irreducible Error.

I wonder what is the difference between reducing bias and reducing reducible error? I thought one always implies another.

Answer 1

Reducible error is composed of bias and variance of the estimator. While reducing the bias you are generally increasing the variance which may result in an increased reducible error.

Answer 2

Expected Test Error is given by

  E(error) = Squared Bias + Variance + noise

where noise [Var(error)] is irreducible error. So, you may also write

  Reducible error = Squared Bias + Variance

By changing linear model to quadratic one, you are reducing Bias of the model which generally increases the variance (variance due to training sample). For selecting between the model, you need to find a trade-off between Bias and Variance.

If reduction in bias (due to introducing quadratic term) is more than increase in variance, then quadratic model would be better.