What is the relationship between estimation error, approximation error, bias, variance in machine learning?

Question

I'm a beginner in machine learning. I was reading http://ciml.info/ 5.9 Bias/Variance Trade-off

According to this book:

The trade-off between estimation error and approximation error is often called the bias/variance trade-off, where “approximation error” is “bias” and “estimation error” is “variance.”

However, I have done some extra research like this:

http://scott.fortmann-roe.com/docs/BiasVariance.html

It gives a very different definition of bias/variance. According to the book, bias is the error of the best classifier. According to the blog, bias is the expected output minus the true output. I just get confused at these two definitions. How are approximation error, bias, estimation error, variance related? Why are there two very different definitions? Thanks a lot!

Answer 1

They are both essentially saying the same thing. Your book takes a more bayesian way of approaching the tradeoff, while the other is a more statistical way of viewing it. There is also the Machine Learning scope which refers to them as the overfitting and underfitting.

I'm guessing you're looking for something like this post which ties the two together.

If you're looking at understanding what the tradeoff essentially is the second link you posted doesn't offer a bad explanation. I'd also like to offer this notebook which presents a simple graphical example of the tradeoff so that you can see it in practice