14
$\begingroup$

Suppose I train a neural network on dataset A and evaluate on dataset B (that has a different feature distribution than dataset A). If I increase the amount of data in dataset A by a factor of 10, is it likely to decrease accuracy on dataset B?

$\endgroup$
  • 4
    $\begingroup$ "that has a different feature distribution than dataset A" Depending on what you mean by this, if the distribution of training set is different than the test set, that's not an overfitting issue, it is either functional shift or covariate shift. $\endgroup$ – Cagdas Ozgenc Nov 14 at 7:20
  • 1
    $\begingroup$ It seems like you may have something akin to a transfer learning problem. $\endgroup$ – eric_kernfeld Nov 14 at 19:06
  • $\begingroup$ It helps if you state which type of NN. Some are more prone to overfit than others. As stands this is an incredibly broad question. $\endgroup$ – smci Nov 14 at 21:15
12
$\begingroup$

On the contrary, more data is almost always better at generalizing to unseen data. The more examples of the data-generating process, the closer the model predictions will get to that of the population. After all, your model has seen a larger part of the population.

Hypothetically, if all hyperparameters were to be held constant, then more data means more steps along the gradient at the same learning rate, which could indeed overfit more easily. However, if you regularize appropriately, choose the right learning rate, etc., then this isn't a problem.

That said, if the new and old data do not come from the same distribution, then simply adding more data will not remedy this. You should probably look into over-/undersampling, or other methods, depending on what exactly you mean by a different feature distribution.

$\endgroup$
  • 1
    $\begingroup$ In my opinion for a consistent learning algorithm more data should always lead a lower generalization error at the same probability level. I think you are thinking of cases where learning is not let to converge. This is more of an implementation issue rather than the facts related to the question. $\endgroup$ – Cagdas Ozgenc Nov 14 at 11:19
  • $\begingroup$ @CagdasOzgenc Perhaps you misread my second paragraph - I completely agree with you. I've made a minor edit to hopefully make it clearer. $\endgroup$ – Frans Rodenburg Nov 14 at 11:36
  • $\begingroup$ In agreement with CagdasOzgenc, I think the formulations of the answer might be misleading/confusing. Convergence seems more like a side issue to me. As long as the sample is representative, having a larger one would improve the ability of a neural network to fit the signal rather than the noise, as the information of the signal accumulates with sample size while noise does not. $\endgroup$ – Richard Hardy Nov 14 at 13:01
  • $\begingroup$ A little better, but not all that much. You do not elaborate on why more data is better; instead, you explain why it can be worse. (This is only my opinion, of course.) $\endgroup$ – Richard Hardy Nov 14 at 13:13
  • $\begingroup$ Thanks, I think it is an improvement. $\endgroup$ – Richard Hardy Nov 14 at 14:07
3
$\begingroup$

The fact that dataset B "has a different feature distribution than dataset A" makes the question fairly ambiguous. It's equivalent to asking how well will a neural net trained for problem A work on problem B - there is no definitive answer. But yes, it is possible that training on more samples from dataset A will make your neural net perform worse on dataset B.

One example where this may come up is if you're training your neural net on simulated data and validating on "real world" data. Because the simulated data doesn't perfectly represent the real data, the neural net may learn patterns in the simulated data that don't generalize to the real world. In that case there is likely to be a training set size that optimizes your performance on the validation set and additional training points will reduce the validation accuracy. This is not a great way to go about things though.

This issue is not what overfitting typically refers to, but it does have an analogous nature (perhaps someone else can help with an exact term for this).

$\endgroup$
0
$\begingroup$

Yes it can. One possibility is that all the examples in dataset A are similar, so your classifier may be overfitting this dataset, and may work worse on dataset B. It might well be your case, if you are fitting the features very common in A but nearly absent in B.

Another possibility is that you are overfitting the same features of A and B. Adding more samples to A will make this dataset less biased, but the classifier will work worse on B.

This is not specific to neural networks.

$\endgroup$
0
$\begingroup$

Some theoretical considerations.

The book "Probabilistic Learning" states that "Neural networks with one hidden layer are universally consistent if the parameters are well-chosen." They mean that, as training set size goes to infinity, the error rate converges to error of the Bayes classifier.

The book "Understanding machine learning" gives an estimate of VC-dimension of the class of hypotheses of NN. Using Fundamental Theorem of Learning may give an idea, how big shall be a training set to get the accuracy you want. Usually, it is huge.

Neither of these results mean that if you increase training set 10 times, the accuracy will be better. It only mean that if you increase your training set indefinitely, then, eventually, the results will get better. But then, they do not say how to select the "well - chosen" parameters. So, yes, increasing size of data 10 times may get worse results from theoretical point of view.

$\endgroup$
0
$\begingroup$

The data in your training and test sets can be modeled as h(x) + noise. In this context, the noise is the variability in your training and test data that is not explained by some common (theoretically optimal) model h(x). The important thing here is that, for example, if your training and test sets are sampled from entirely different distributions, then ALL of your data is noise, even if on their own, both training and test set data are very well structured. In this case, even a model with 1 or 2 parameters will be overfitting right away - regardless of how many data points you have in your training set!

In other words - the greater the amount of noise in your data, the easier it is to overfit and the simpler model you are restricted to using. With, say, gaussian noise, increasing the amount of data in your training set increases the data-to-noise ratio, reducing overfitting. If your training and test data are from (slightly) different distributions, increasing the amount of data will do nothing to reduce this source of noise! The data-to-noise ratio will stay the same. Only other sources of noise will be eliminated (e.g. measurement noise, if applicable).

So increasing the amount of data can only make overfitting worse if you mistakenly also increase the complexity of your model. Otherwise, the performance on the test set should improve or remain the same, but not get significantly worse.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.