What impact does increasing the training data have on the overall system accuracy?

Question

Can someone summarize for me with possible examples, at what situations increasing the training data improves the overall system? When do we detect that adding more training data could possibly over-fit data and not give good accuracies on the test data?

This is a very non-specific question, but if you want to answer it specific to a particular situation, please do so.

Answer 1

In most situations, more data is usually better. Overfitting is essentially learning spurious correlations that occur in your training data, but not the real world. For example, if you considered only my colleagues, you might learn to associate "named Matt" with "has a beard." It's 100% valid ($n=4$, even!), but it's obviously not true in general. Increasing the size of your data set (e.g., to the entire building or city) should reduce these spurious correlations and improve the performance of your learner.

That said, one situation where more data does not help---and may even hurt---is if your additional training data is noisy or doesn't match whatever you are trying to predict. I once did an experiment where I plugged different language models[*] into a voice-activated restaurant reservation system. I varied the amount of training data as well as its relevance: at one extreme, I had a small, carefully curated collection of people booking tables, a perfect match for my application. At the other, I had a model estimated from huge collection of classic literature, a more accurate language model, but a much worse match to the application. To my surprise, the small-but-relevant model vastly outperformed the big-but-less-relevant model.

---

A surprising situation, called double-descent, also occurs when size of the training set is close to the number of model parameters. In these cases, the test risk first decreases as the size of the training set increases, transiently increases when a bit more training data is added, and finally begins decreasing again as the training set continues to grow. This phenomena was reported 25 years in the neural network literature (see Opper, 1995), but occurs in modern networks too (Advani and Saxe, 2017). Interestingly, this happens even for a linear regression, albeit one fit by SGD (Nakkiran, 2019). This phenomenon is not yet totally understood and is largely of theoretical interest: I certainly wouldn't use it as a reason not to collect more data (though I might fiddle with the training set size if n==p and the performance were unexpectedly bad).

---

[*]A language model is just the probability of seeing a given sequence of words e.g. $P(w_n = \textrm{'quick', } w_{n+1} = \textrm{'brown', } w_{n+2} = \textrm{'fox'})$. They're vital to building halfway decent speech/character recognizers.

---

Answer 2

One note: by adding more data (rows or examples, not columns or features) your chances of overfitting decrease rather than increase.

The two paragraph summary goes like this:

• Adding more examples, adds diversity. It decreases the generalization error because your model becomes more general by virtue of being trained on more examples.
• Adding more input features, or columns (to a fixed number of examples) may increase overfitting because more features may be either irrelevant or redundant and there's more opportunity to complicate the model in order to fit the examples at hand.

There are some simplistic criteria to compare quality of models. Take a look for example at AIC or at BIC.

They both show that adding more data always makes models better, while adding parameter complexity beyond the optimum, reduces model quality.

Answer 3

Increasing the training data always adds information and should improve the fit. The difficulty comes if you then evaluate the performance of the classifier only on the training data that was used for the fit. This produces optimistically biased assessments and is the reason why leave-one-out cross validation or bootstrap are used instead.

Answer 4

Ideally, once you have more training examples you’ll have lower test-error (variance of the model decrease, meaning we are less overfitting), but theoretically, more data doesn’t always mean you will have more accurate model since high bias models will not benefit from more training examples.

See here: In Machine Learning, What is Better: More Data or better Algorithms

High-variance – a model that represent training set well, but at risk of overfitting to noisy or unrepresentative training data.

High bias – a simpler model that doesn’t tend to overfit, but may underfit training data, failing to capture important regularities.

Answer 5

The spectrum analysis will help in the analysis of the diversity of the sample, in fact, the false information will be learned in the modelling if not "real-samples" added, which usually called the over-fitting. Usually, if the provided information by sample is less, the more real sample is encouraged to be provided to ensure the useful information can be used in the testing. Good luck!