What impact does increasing the training data have on the overall system accuracy?

Can someone summarize for me with possible examples, at what situations increasing the training data improves the overall system? When do we detect that adding more training data could possibly over-fit data and not give good accuracies on the test data?

This is a very non-specific question, but if you want to answer it specific to a particular situation, please do so.

-

Practical concerns like memory and processor time aside, I can't imagine any situation where having more representative training data leads to a worse outcome. Overfitting is essentially learning spurious correlations that occur in your training data, but not the real world. For example, if you considered only my colleagues, you might learn to associate "named Matt" with "has a beard." It's 100% valid ($n=4$, even!), but it's obviously not true in general. Increasing the size of your data set (e.g., to the entire building or city) should reduce these spurious correlations and improve the performance of your learner.

That said, one situation where more data does not help---and may even hurt---is if your additional training data is noisy or doesn't match whatever you are trying to predict. I once did an experiment where I plugged different language models[*] into a voice-activated restaurant reservation system.

I varied the amount of training data as well as its relevance: at one extreme, I had a small, carefully curated collection of people booking tables, a perfect match for my application. At the other, I had a model estimated from huge collection of classic literature, a more accurate language model, but a much worse match to the application. To my surprise, the small-but-relevant model vastly outperformed the big-but-less-relevant model.

Unfortunately, I don't think there are any hard and fast rules for this sort of trade-off. You'll have to try it and see how it works.

[*]A language model is just the probability of seeing a given sequence of words e.g. $P(w_n = \textrm{'quick', } w_{n+1} = \textrm{'brown', } w_{n+2} = \textrm{'fox'})$. They're vital to building halfway decent speech/character recognizers.

-

Increasing the training data always adds information and should improve the fit. The difficulty comes if you then evaluate the performance of the classifier only on the training data that was used for the fit. This produces optimistically biased assessments and is the reason why leave-one-out cross validation or bootstrap are used instead.

-

One note: by adding more data (rows or examples, not columns or features) your chances of overfitting decrease rather than increase.

The two paragraph summary goes like this:

• Adding more examples, adds diversity. It decreases the generalization error because your model becomes more general by virtue of being trained on more examples.
• Adding more input features, or colums (to a fixed number of examples) may increase overfitting because more features may be either irrelevant or redundant and there's more opportunity to complicate the model in order to fit the examples at hand.

There are some simplistic criteria to compare quality of models. Take a look for example at AIC or at BIC.

They both show that adding more data always makes models better, while adding parameter complexity beyond the optimum, reduces model quality.

-