I use Keras to train a Neural Network. The input of the network is a set of 2d points. The count of inputs is fixed. E.g. 5 points are used as input.
Every 10th iteration I do not only train the network, but i also use some validation data to validate it. Both data records (training + validation) are always generated with the same function (just some random placed 2d Gaussian distributed clusters).
My problem is that the network always has a significant lower training loss than a validation loss (e.g. ~10% lower). This means the network overfits. But this should not be possible if I generate data? Or is there a reason why this is still possible (maybe Keras does something special)?
Currently I do not use Dropout, because I thought it is impossible that the networks overfits if the data is always randomly generated.