I'm a beginner in machine learning. I was reading http://ciml.info/ 5.9 Bias/Variance Trade-off

According to this book:

The trade-off between estimation error and approximation error is often called the bias/variance trade-off, where “approximation error” is “bias” and “estimation error” is “variance.”

However, I have done some extra research like this:


It gives a very different definition of bias/variance. According to the book, bias is the error of the best classifier. According to the blog, bias is the expected output minus the true output. I just get confused at these two definitions. How are approximation error, bias, estimation error, variance related? Why are there two very different definitions? Thanks a lot!


1 Answer 1


They are both essentially saying the same thing. Your book takes a more bayesian way of approaching the tradeoff, while the other is a more statistical way of viewing it. There is also the Machine Learning scope which refers to them as the overfitting and underfitting.

I'm guessing you're looking for something like this post which ties the two together.

If you're looking at understanding what the tradeoff essentially is the second link you posted doesn't offer a bad explanation. I'd also like to offer this notebook which presents a simple graphical example of the tradeoff so that you can see it in practice


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.