5
$\begingroup$

I am training a large set of neural networks for a quite simple task. 10 of the networks have the same configuration, but have different amount of data. The 10 networks each have one hidden layer, with 2 neurons in it. The first network gets 1.000 training examples, the next 2.000 and the last one gets 10.000 training examples.

The 1000 training examples for for the first network, is a subset of the 2000 training examples for the second network, which again is a subset of the 3000 training examples for the third network and so forth.

I train my networks using the build in NN-Toolbox in MATLAB, where I use the Levenberg-Marquardt algorithm. When I train the networks, they all end up with a mean-squared error around 0.007 (which seems fair for my particular problem). The only one which differs, is the first one, which achieves a MSE of around 0.002.

After I have trained my networks on the data, I test them on some test data which I did not use for training. Now the problem is, that the network with 1000 training examples seriously outperforms the other networks. From the MSE-score, this seems fair, but I do not understand why the networks with more available data, and even the data which network one uses, fails to learn the parameters better.

Is there such a thing as too much data?

$\endgroup$
  • $\begingroup$ Note also that neural networks are generally not convex, which means that the final result can depend strongly on the initial conditions. In the Matlab neural network toolbox, the initial conditions are chosen randomly, so your results may be strongly influenced by random chance. $\endgroup$ – Daniel Golden Jan 28 '14 at 16:28
1
$\begingroup$

It sounds like the networks you train with more data overfit to that data and hence perform worse on different data. The concept the networks are supposed to learn may be obvious from the small data set, but adding more data in the other set obscures it (or even transforms it into a different concept).

One way to mitigate this effect is to make sure that the distribution of the different predictions is approximately the same in the training and test data (stratification). Alternatively, you could train and evaluate your networks using cross validation.

$\endgroup$
  • 2
    $\begingroup$ What I found out, was that I actually had some errors in my input data. The result of this, was that the networks with a lot of data, actually learned to respect these errors, which the networks with only very little data did not experience in the same amount. On top of that, the data became very hard to learn, and the results I got were very random. $\endgroup$ – utdiscant Apr 25 '12 at 11:21
  • $\begingroup$ A valuable finding indeed! $\endgroup$ – jbowman Apr 25 '12 at 17:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.