Neural Network with artificial data overfits?

Question

I use Keras to train a Neural Network. The input of the network is a set of 2d points. The count of inputs is fixed. E.g. 5 points are used as input.

Every 10th iteration I do not only train the network, but i also use some validation data to validate it. Both data records (training + validation) are always generated with the same function (just some random placed 2d Gaussian distributed clusters).

My problem is that the network always has a significant lower training loss than a validation loss (e.g. ~10% lower). This means the network overfits. But this should not be possible if I generate data? Or is there a reason why this is still possible (maybe Keras does something special)?

Currently I do not use Dropout, because I thought it is impossible that the networks overfits if the data is always randomly generated.

Thank you

Answer 1

Neural networks could "learn" where the labels are completely random. In this case nets will simply memorize the label for each sample thus giving you low training error but high testing error.

There is an interesting paper on this topic where authors trained nets on ImageNet, but corrupted the label to be random. What they found was that is the net has enough parameters, but labels are completely random it can still memorize all of them.