Can I have too much data?

Question

I am training a large set of neural networks for a quite simple task. 10 of the networks have the same configuration, but have different amount of data. The 10 networks each have one hidden layer, with 2 neurons in it. The first network gets 1.000 training examples, the next 2.000 and the last one gets 10.000 training examples.

The 1000 training examples for for the first network, is a subset of the 2000 training examples for the second network, which again is a subset of the 3000 training examples for the third network and so forth.

I train my networks using the build in NN-Toolbox in MATLAB, where I use the Levenberg-Marquardt algorithm. When I train the networks, they all end up with a mean-squared error around 0.007 (which seems fair for my particular problem). The only one which differs, is the first one, which achieves a MSE of around 0.002.

After I have trained my networks on the data, I test them on some test data which I did not use for training. Now the problem is, that the network with 1000 training examples seriously outperforms the other networks. From the MSE-score, this seems fair, but I do not understand why the networks with more available data, and even the data which network one uses, fails to learn the parameters better.

Is there such a thing as too much data?

Answer 1

It sounds like the networks you train with more data overfit to that data and hence perform worse on different data. The concept the networks are supposed to learn may be obvious from the small data set, but adding more data in the other set obscures it (or even transforms it into a different concept).

One way to mitigate this effect is to make sure that the distribution of the different predictions is approximately the same in the training and test data (stratification). Alternatively, you could train and evaluate your networks using cross validation.