31
$\begingroup$

Can someone summarize for me with possible examples, at what situations increasing the training data improves the overall system? When do we detect that adding more training data could possibly over-fit data and not give good accuracies on the test data?

This is a very non-specific question, but if you want to answer it specific to a particular situation, please do so.

$\endgroup$
2
  • $\begingroup$ just wondering - is this about whether a 50-50 split into train/test is better than say 75-25? $\endgroup$ Dec 18, 2019 at 2:15
  • $\begingroup$ If you want to approximate a function of the form $ax^2+bx+c$, then using 3 data points is definitely better than 2, 5 is better than 3 and 10 is better than 5. However, if you want to approximate a function of the form $ax+b$, provided your training data is somewhat representative, a sample size of 100 would barely better than of 5. $\endgroup$
    – Tran Khanh
    Dec 23, 2023 at 10:09

5 Answers 5

39
$\begingroup$

In most situations, more data is usually better. Overfitting is essentially learning spurious correlations that occur in your training data, but not the real world. For example, if you considered only my colleagues, you might learn to associate "named Matt" with "has a beard." It's 100% valid ($n=4$, even!) when considering only the small group of people working on floor, but it's obviously not true in general. Increasing the size of your data set (e.g., to the entire building or city) should reduce these spurious correlations and improve the performance of your learner.

That said, one situation where more data does not help---and may even hurt---is if your additional training data is noisy or doesn't match whatever you are trying to predict. I once did an experiment where I plugged different language models[*] into a voice-activated restaurant reservation system. I varied the amount of training data as well as its relevance: at one extreme, I had a small, carefully curated collection of people booking tables, a perfect match for my application. At the other, I had a model estimated from huge collection of classic literature, a more accurate language model, but a much worse match to the application. To my surprise, the small-but-relevant model vastly outperformed the big-but-less-relevant model.


A surprising situation, called **double-descent**, also occurs when size of the training set is close to the number of model parameters. In these cases, the test risk first decreases as the size of the training set increases, transiently *increases* when a bit more training data is added, and finally begins decreasing again as the training set continues to grow. This phenomena was reported 25 years in the neural network literature (see Opper, 1995), but occurs in modern networks too ([Advani and Saxe, 2017][1]). Interestingly, this happens even for a linear regression, albeit one fit by SGD ([Nakkiran, 2019][2]). This phenomenon is not yet totally understood and is largely of theoretical interest: I certainly wouldn't use it as a reason not to collect more data (though I might fiddle with the training set size if n==p and the performance were unexpectedly bad).
[*]A language model is just the probability of seeing a given sequence of words e.g. $P(w_n = \textrm{'quick', } w_{n+1} = \textrm{'brown', } w_{n+2} = \textrm{'fox'})$. They're vital to building halfway decent speech/character recognizers.
$\endgroup$
3
  • $\begingroup$ Thanks so I guess quality of data is what matters the most. On the second block "when size of training set is close to size of parameters" size of training set implies number of rows and size of parameters refer to number of features? $\endgroup$
    – haneulkim
    Mar 17, 2022 at 13:07
  • $\begingroup$ Overall "quality" but also the match between your training data and the eventual application: I'm sure GTP-3 (etc) would work well here too, but it's orders of magnitude bigger. $\endgroup$ Mar 19, 2022 at 16:38
  • $\begingroup$ @MattKrause Out of curiosity, what was the performance of each data group (the small, carefully curated collection of people vs the large collection) on the training error? You used the word 'outperform', but I would be interested in more context. I would especially like to know if the model with more data overfit or not? $\endgroup$
    – kklaw
    Jan 16, 2023 at 13:52
16
$\begingroup$

It is all data-dependent: added example features can span the range between good-features and pure-noise.

Let's assume some grounded in reality (supposedly useful) & randomly selected data.

By adding more data (rows or examples, not columns or features) your chances of over-fitting decrease rather than increase.

All other things being equal, the two paragraph summary goes like this:

  • Adding more examples, adds diversity. It decreases (improves) the generalization error because your model becomes more general by virtue of being trained on more examples.
  • Adding more input features, or columns to a fixed number of examples may work both ways. Good features can help up to a certain point. But once overdone, they increase over-fitting. More features may be either irrelevant or redundant and there's more opportunity to complicate the model in order to fit the fixed-number of examples at hand.

There are some simplistic criteria to compare quality of models. Take a look for example at AIC or at BIC.

They both show that adding more data always makes models better, while adding parameter complexity beyond the optimum, reduces model quality.

If you have the luxury of a large or unlimited number of examples, there's a practical way to optimize the modeling process: watch the testing (out-of-sample) error rate and make sure to stop training once the out-of-sample error rates bottom out. While doing that, also watch the training error rate. If the smoothed/moving-average rate fails to keep converging, you may have non-stationary data, or bad/irrelevant features or representation (e.g. the NN layout) to begin with.

$\endgroup$
2
  • 1
    $\begingroup$ I think you're using "better" in two different senses here. In the first bullet, most would agree that a more generalizable model is better. In the second, by more data you are as you say referring to more features. So adding more features should always make the model better. I am not sure that is true. If they are pure noise (or biased systematically) then how can that automatically mean a model? If you use the AIC, BIC or bayesian evidence Z then you could say a model with more evidence for it is "better", but that's a function of the data [Z(\vec{d})], so can go up as well as down. $\endgroup$
    – jtlz2
    Dec 21, 2023 at 11:38
  • 1
    $\begingroup$ Thanks @jtlz2. I rephrased some sentences to make the answer more accurate. $\endgroup$
    – arielf
    Dec 23, 2023 at 2:51
2
$\begingroup$

Increasing the training data always adds information and should improve the fit. The difficulty comes if you then evaluate the performance of the classifier only on the training data that was used for the fit. This produces optimistically biased assessments and is the reason why leave-one-out cross validation or bootstrap are used instead.

$\endgroup$
1
  • $\begingroup$ I disagree that information is always added in the information-theoretic sense - it could be noise, which unless it matches reality is not signal. Surely it's the test data that are the final arbiter of any fit improvement, so the question is how aligned the two sets are. Agree re your other points! $\endgroup$
    – jtlz2
    Dec 21, 2023 at 11:41
2
$\begingroup$

Ideally, once you have more training examples you’ll have lower test-error (variance of the model decrease, meaning we are less overfitting), but theoretically, more data doesn’t always mean you will have more accurate model since high bias models will not benefit from more training examples.

See here: In Machine Learning, What is Better: More Data or better Algorithms

High-variance – a model that represent training set well, but at risk of overfitting to noisy or unrepresentative training data.

High bias – a simpler model that doesn’t tend to overfit, but may underfit training data, failing to capture important regularities.

$\endgroup$
1
  • $\begingroup$ Spot on as it recognizes that more data doesn't necessarily mean a better model (somehow defined!). $\endgroup$
    – jtlz2
    Dec 21, 2023 at 11:43
1
$\begingroup$

I agree with @Serendipity:

The performance of neural networks can continually improve as more and more data is provided to the model, BUT the capacity of the model must be adjusted to support the increases in data.

Let's say that you have a very small object detection model (7.2M parameters), it won't be able to learn all the information that you feed to it as the weights won't be able to accommodate for all the possible distributions found in the data, given that the data is complex enough.

The model will only be able to learn large data variability if its capacity makes it possible

$\endgroup$
3
  • $\begingroup$ How are you defining "capacity" - number of parameters? Nice! $\endgroup$
    – jtlz2
    Dec 21, 2023 at 11:42
  • 1
    $\begingroup$ @jtlz2. Capacity is the model's ability to learn a wide variety of data $\endgroup$
    – Mikel B
    Dec 22, 2023 at 8:09
  • 1
    $\begingroup$ I would say it is more a matter of architecture choices more than number of parameters. For example: type of layers used, feature aggregation methods, regularization techniques, activation functions... $\endgroup$
    – Mikel B
    Dec 22, 2023 at 8:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.