9
$\begingroup$

It is clear that more training data will help lower the variance of a high variance model since there will be less overfitting if the learning algorithm is exposed to more data samples.

However, what impact does training data size have on a high bias model? Generally, will more training data lower the bias, will it have no effect, or will it cause a further increase in the bias?

This question is more specific than the following question which is similar: What impact does increasing the training data have on the overall system accuracy?

One of the answers actually says that "high bias models will not benefit from more training examples". But there does not seem to be any consensus.

$\endgroup$

2 Answers 2

8
$\begingroup$

However, what impact does training data size have on a high bias model? Generally, will more training data lower the bias, will it have no effect, or will it cause a further increase in the bias?

You mean a model with prediction errors due to high bias?

Bias, is defined as $\operatorname{Bias}[\hat{f}(x)]=\mathrm{E}[\hat{f}(x)]-f(x)$ and thus would not be affected by increasing the training set size. If your model predicts vastly different values when the training set changes, i.e., if the error is largely defined by the variance of the predictions, than you can improve the overall loss by more training data, because the model will learn to generalize better, and hence the variance term will go down. To decrease the bias term, you probably need to choose a different model.

$\endgroup$
3
  • 1
    $\begingroup$ That makes sense, thank you everyone for your contributions! $\endgroup$
    – Ali
    Commented May 19, 2019 at 20:23
  • $\begingroup$ Hmm, I'm confused by this answer. If you increase the training data set, $f(x)$ doesn't change, but $E[\hat{f}(x)]$ should change because having additional data should change your estimator? If so, isn't bias affected? $\endgroup$
    – David
    Commented Jun 23, 2020 at 21:51
  • $\begingroup$ f^(x) will change but E[f^(x)] does not change actually. x affects f^(x) because you are looking at a specific x. E[f^(x)] you are averaging out all X so the resulting quantity is not a function of the training data anymore. $\endgroup$
    – Jing
    Commented Apr 1, 2021 at 3:24
-3
$\begingroup$

1- Increasing training data size leads to decrease variance. 2- Decrease the variance leads to increase the bias. so increase the training data size leads to decrease variance and increase variance.

$\endgroup$
2
  • $\begingroup$ This only makes sense if you a) assume that the loss doesn't improve with more training data and b) that you are talking about the squared difference between the prediction and targets and this doesn't universally apply for 0/1 loss etc. $\endgroup$
    – resnet
    Commented May 6, 2019 at 5:22
  • 2
    $\begingroup$ This just strikes me as incorrect. If your data is an iid sample, then a larger sample will decrease variance, and keep bias exactly the same. $\endgroup$ Commented May 6, 2019 at 5:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.