I am training a large set of neural networks for a quite simple task. 10 of the networks have the same configuration, but have different amount of data. The 10 networks each have one hidden layer, with 2 neurons in it. The first network gets 1.000 training examples, the next 2.000 and the last one gets 10.000 training examples.
The 1000 training examples for for the first network, is a subset of the 2000 training examples for the second network, which again is a subset of the 3000 training examples for the third network and so forth.
I train my networks using the build in NN-Toolbox in MATLAB, where I use the Levenberg-Marquardt algorithm. When I train the networks, they all end up with a mean-squared error around 0.007 (which seems fair for my particular problem). The only one which differs, is the first one, which achieves a MSE of around 0.002.
After I have trained my networks on the data, I test them on some test data which I did not use for training. Now the problem is, that the network with 1000 training examples seriously outperforms the other networks. From the MSE-score, this seems fair, but I do not understand why the networks with more available data, and even the data which network one uses, fails to learn the parameters better.
Is there such a thing as too much data?