Training set error as an estimate of bias

Question

In his machine learning lectures (1 min 30 sec), Andrew Ng seems to estimate the bias using the training set error. Why is it ok to do it?

The definition of "bias" in machine learning (see wiki or any references from it), at least for the mean squared error cost function, is the expected error when using as a prediction the expected prediction made by various training sets. So why would the error (for a given point) for a single training set be anywhere close to the error (for that point) when using the average of all possible training sets?

Answer 1

Andrew doesn't use the training set error to estimate bias as such. He compares training set error and cross-validated (or test set) error to figure out whether the error of his model is dominated by its bias component or its variance component, which see bias–variance tradeoff.

If most of your error comes from bias, you should probably examine more flexible methods. For e.g. if you're trying to fit a plane to something super wavy, which would be underfitting. If most of your error comes from variance, you're probably using a too-flexible method. For e.g. if you're fitting a super wavy function to something that is really better approximated by a plane, which would be overfitting.

In the first case above you won't be able to fit either the training set or the test set very well, so both of those errors will be high; in the second case you will be able to fit the training set perfectly but will have a poor fit in the test set, so training error will be low, but test error will be high.

Answer 2

This is not a 100% confident answer but my thoughts on this question.

1. I have the same confusion as @max regarding Andrew Ng's discussion on bias/variance and training error/testing error. I suppose @einar is correct - Andrew doesn't use the training set error to estimate bias. Instead, Andrew compares training set and test set error to figure out whether the error of the model is dominated by bias or variance. But the side effect is, it may cause confusion for beginners (for example, me, when I first attended Andrew's machine learning module on Coursera).
2. To estimate the bias and variance of the machine learning model, my recommendation is to use the mlxtend library, which comes with some good tutorials, such as this. Essentially, the bias_variance_decomp function uses a bootrapping function to sample the training data for multiple rounds, and in each round, the sampled training data is used to train a model, which is then applied to make predictions on the given and fixed testing data. Based on this process, the bias of the algorithm is estimated as the average prediction of the testing data (across multiple rounds), and the variance is estimated as the variance of predictions across multiple rounds. These two estimations are consistent with the theory of bias and variance.

Any comments or critiques are highly welcome!