What is the relationship between estimation error, approximation error, bias, variance in machine learning?

Question

I'm a beginner in machine learning. I was reading http://ciml.info/ 5.9 Bias/Variance Trade-off

According to this book:

The trade-off between estimation error and approximation error is often called the bias/variance trade-off, where “approximation error” is “bias” and “estimation error” is “variance.”

However, I have done some extra research like this:

http://scott.fortmann-roe.com/docs/BiasVariance.html

It gives a very different definition of bias/variance. According to the book, bias is the error of the best classifier. According to the blog, bias is the expected output minus the true output. I just get confused at these two definitions. How are approximation error, bias, estimation error, variance related? Why are there two very different definitions? Thanks a lot!

Answer 1

They are both essentially saying the same thing. Your book takes a more bayesian way of approaching the tradeoff, while the other is a more statistical way of viewing it. There is also the Machine Learning scope which refers to them as the overfitting and underfitting.

I'm guessing you're looking for something like this post which ties the two together.

If you're looking at understanding what the tradeoff essentially is the second link you posted doesn't offer a bad explanation. I'd also like to offer this notebook which presents a simple graphical example of the tradeoff so that you can see it in practice

Answer 2

The book, and the post above, are not quite correct.

In general, bias is not approximation error, and variance is not estimation error. It is not even the case that one is a special case of the other, as is sometimes stated. The true relation is more subtle. The approximation error is in fact just one component of the bias, and, the estimation error contributes to both bias and variance.

Bias = (Approximation Error) + (Estimation Bias)

Variance = (Optimisation Error) + (Estimation Variance)

And furthermore…

Estimation error = (estimation bias) + (estimation variance)

The optimisation error comes from the performance of the learning algorithm. The reason for two definitions is simply that they came from different research communities at different times. The BV is from statistics, and the AE from computational learning theory. The motivation for the AE was to define generalisation bounds as a function of model capacity and the size of the data.

The primary advantage of the bias/variance terminology is that the terms can be estimated from data, which is not possible with approximation/estimation. A disadvantage is that the bias is clearly a flawed proxy for model capacity, since the estimation bias is in there too. To further complicate things, the estimation bias can be negative in some cases.

In the very special case of unregularized linear regression, i.e., OLS, then the two decompositions are equivalent, but not in general.

To understand fully requires more detail than is possible in this reply, but is explained in the paper linked below.

“Bias/Variance is not the same as Approximation/Estimation” Transactions on Machine Learning Research, February 2024. https://openreview.net/pdf?id=4TnFbv16hK

———

Full disclosure: I’m an author.