0
$\begingroup$

True or false:

The more data examples we have, the more confidence we will have in using a high-capacity model without fear of overfitting.

Here the “capacity” of a machine learning algorithm corresponds, informally, to the “size” or “richness” or “complexity” of the considered set of functions among which it searches for the best prediction function.

$\endgroup$
0
$\begingroup$

True (EDIT: assuming the extra training data is a random sample drawn from the population, and not, for example, the result of resampling on a smaller set). A model that tends to overfit because of its high capacity has high variance, i.e. its parameter estimates vary a lot based on the training data sample you use (because the model is capable of precisely fitting any small data set).

The variance of parameter estimates is reduced when you increase the size of the training data, making the problem of overfitting less severe. With more training data, the benefits of using a high-capacity model may start to outweigh the drawbacks (high variance), which are reduced.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ It is difficult to see how this answer can be generally correct, because it includes no consideration of how the "data examples" were obtained. As an extreme example, some people in this forum have proposed simply replicating the data they have in order to enlarge their datasets. Obviously that shouldn't increase our confidence in any fitting procedure. $\endgroup$ – whuber Oct 29 '19 at 14:40
  • 1
    $\begingroup$ You're absolutely correct, my answer rests on the unstated assumption that having more data examples means that more data was randomly sampled from the population, not generated artificially, for example. $\endgroup$ – Vincent B. Lortie Oct 29 '19 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.