Can someone summarize for me with possible examples, at what situations increasing the training data improves the overall system? When do we detect that adding more training data could possibly over-fit data and not give good accuracies on the test data?
This is a very non-specific question, but if you want to answer it specific to a particular situation, please do so.
Practical concerns like memory and processor time aside, I can't imagine any situation where having more representative training data leads to a worse outcome. Overfitting is essentially learning spurious correlations that occur in your training data, but not the real world. For example, if you considered only my colleagues, you might learn to associate "named Matt" with "has a beard." It's 100% valid ($n=4$, even!), but it's obviously not true in general. Increasing the size of your data set (e.g., to the entire building or city) should reduce these spurious correlations and improve the performance of your learner.
That said, one situation where more data does not help---and may even hurt---is if your additional training data is noisy or doesn't match whatever you are trying to predict. I once did an experiment where I plugged different language models[*] into a voice-activated restaurant reservation system.
I varied the amount of training data as well as its relevance: at one extreme, I had a small, carefully curated collection of people booking tables, a perfect match for my application. At the other, I had a model estimated from huge collection of classic literature, a more accurate language model, but a much worse match to the application. To my surprise, the small-but-relevant model vastly outperformed the big-but-less-relevant model.
Unfortunately, I don't think there are any hard and fast rules for this sort of trade-off. You'll have to try it and see how it works.
[*]A language model is just the probability of seeing a given sequence of words e.g. $P(w_n = \textrm{'quick', } w_{n+1} = \textrm{'brown', } w_{n+2} = \textrm{'fox'})$. They're vital to building halfway decent speech/character recognizers.
One note: by adding more data (rows or examples, not columns or features) your chances of overfitting decrease rather than increase.
The two paragraph summary goes like this:
There are some simplistic criteria to compare quality of models. Take a look for example at AIC or at BIC.
They both show that adding more data always makes models better, while adding parameter complexity beyond the optimum, reduces model quality.
Increasing the training data always adds information and should improve the fit. The difficulty comes if you then evaluate the performance of the classifier only on the training data that was used for the fit. This produces optimistically biased assessments and is the reason why leave-one-out cross validation or bootstrap are used instead.
Ideally, once you have more training examples you’ll have lower test-error (variance of the model decrease, meaning we are less overfitting), but theoretically, more data doesn’t always mean you will have more accurate model since high bias models will not benefit from more training examples.
See here: In Machine Learning, What is Better: More Data or better Algorithms
High-variance – a model that represent training set well, but at risk of overfitting to noisy or unrepresentative training data.
High bias – a simpler model that doesn’t tend to overfit, but may underfit training data, failing to capture important regularities.
The spectrum analysis will help in the analysis of the diversity of the sample, in fact, the false information will be learned in the modelling if not "real-samples" added, which usually called the over-fitting. Usually, if the provided information by sample is less, the more real sample is encouraged to be provided to ensure the useful information can be used in the testing. Good luck!
