0
$\begingroup$

I encountered a question while learning:

While doing a homework assignment, you fit a Linear Model to your data set. You are thinking about changing the Linear Model to a Quadratic one.

  • Using the Quadratic Model will decrease your Irreducible Error
  • (Correct answer) Using the Quadratic Model will decrease the Bias of your model
  • Using the Quadratic Model will decrease the Variance of your model
  • Using the Quadratic Model will decrease your Reducible Error

And an explanation for it:

Introducing the quadratic term will make your model more complicated. More complicated models typically have lower bias at the cost of higher variance. This has an unclear effect on Reducible Error (could go up or down) and no effect on Irreducible Error.

I wonder what is the difference between reducing bias and reducing reducible error? I thought one always implies another.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Presumably you have been given specific definitions for those terms (particularly reducible error, irreducible error). While we might be able to understand the intended meaning in a broad sense, we don't necessarily have exactly those definitions you would have been given. It would be useful context to know exactly how you were expected to interpret them. $\endgroup$
    – Glen_b
    Dec 24, 2015 at 21:48

2 Answers 2

Reset to default
2
$\begingroup$

Reducible error is composed of bias and variance of the estimator. While reducing the bias you are generally increasing the variance which may result in an increased reducible error.

Improve this answer
$\endgroup$
2
$\begingroup$

Expected Test Error is given by 

  E(error) = Squared Bias + Variance + noise

where noise [Var(error)] is irreducible error. So, you may also write

  Reducible error = Squared Bias + Variance

By changing linear model to quadratic one, you are reducing Bias of the model which generally increases the variance (variance due to training sample). For selecting between the model, you need to find a trade-off between Bias and Variance.

If reduction in bias (due to introducing quadratic term) is more than increase in variance, then quadratic model would be better.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.