How does increasing the dataset decrease the cross validation error in case of large variance?

Question

I was recently watching the video lectures by Andrew Ng on Coursera and could not understand the idea as to how increasing the data set lead to decrease in the cross validation error in the case of high variance.Similarly, why is it that a large dataset cannot help decrease the cross validation error in case of high bias?.Can somebody please shed some light on this? Thanks in advance!

Answer 1

Classifiers that display high variance do so because they can overfit the training data--eg, they easily interpret noise as signal. They have high "variance" because, given different sets of training data, the parameters that they estimate will be quite different. Their form, hence, is quite variable.

The more data that you provide, the more difficult it is for a classifier to interpret noise as signal.

Consider a case with 10 variables and 5 examples. Any of the 10 variables might randomly appear to be well-related to the response. By chance, they might go up when the response goes up or vice versa. It only needs to happen five times, and that's not so unlikely.

If you however have 5 million examples, it becomes less likely that 5 million times a variable both goes up with the response and down with the response, by total chance.

A high bias classifier is inflexible--so it might not interpret anything--noise or signal--as signal. Consider the most biased classifier, one that simply outputs 1 for every case, no matter the features. Training this classifier on more data would not change it's output (it learns nothing from the data). As we go further toward the biased end of the spectrum (e.g., lasso with high penalty), this is the case. The error is because it's not paying enough attention to the data, so giving more data won't help.