What is bias in Bias-Variance Tradeoff?

Question

In the book An Introduction to Statistical Learning, chapter 2, it is mentioned that Expected MSE has 3 components:

$E(y_0-\widehat{f}(x_0))^2=Var(\widehat{f}(x_0))+[Bias(\widehat{f}(x_0))]^2+Var(\epsilon )$

My question in this case is about the Bias component, and the text tells that this Bias is due to trying to estimate a complex model using a simpler model $\widehat{f}$, for example trying to fit a linear model to a data with a non-linear underlying best estimate.

My question is what kind of Bias is it? Because Bias I have come across before in statistics context is about data collection/survey when sample is not representing the population because some items in the population have unequal or lower or 0 probability of being included in the sample. And in that context of sampling, bias is either Selection Bias and Response Bias.

Second, why this feature called Bias in the first place? Since Bias is related to sampling due to not representing the population properly.

I am looking for an intuition of this concept.

Answer 1

Unfortunately, this is just another, different concept, also called 'bias.' Its definition is the expectation of the difference between the true, and estimated, parameters.

If there is a non-zero difference, this difference might be because of something like response bias, for example where your estimator is a survey, and that survey makes people answer in a way that's reliably different from the truth -- but the definition here doesn't have any behavioral component to it, necessarily. It's just a mathematical definition, that's also called 'bias.'

For intuition, the example you give is a useful one. When you're using a model that's not complex enough to describe a phenomenon, you'll reliably misestimate the parameters of the true phenomenon - you'll have (mathematical) bias. If you make the model more complicated to compensate though, that model will be harder to estimate, and so you'll get more variance! That's why this equation is so important - it shows you, explicitly, the trade off you're making when you choose a particular model.

If you can think of a better word, I think we'd all be happy to use it!

Answer 2

Generally, bias is when you are tying to estimate something, but because of how you go about estimating the thing, your expected estimate is different from the true value.

Take your sampling example. You are collecting data to attempt to learn something about the population you are sampling from. The sample is bias when it, for some reason, does not correctly reflect the population you're drawing from for purposes of what you are trying to learn. That is, if you took sample after sample in many universes, what you learned from the different samples would still be wrong.

In learning, a model has high bias when averaging over training datasets drawn from many different universes, the model still does not reflect the true process. This tends to happen when the model is underfit. For example, if the truth is a parabola, and you're fitting a line, no matter how many times you train the model or collect more data, you're still going to be wrong.

So, blur your eyes a little, and the concepts are mostly the same.