I'm training a neural network to classify the MNIST digits. For this, I am using this mnist library which states:
It includes 10000 different samples of mnist digits.
Both sets [training and validation] are shuffled, and there are no samples repeated in both sets.
So, what I'm doing is create a 8000 samples training set, and a 2000 validation set, train against the first one, validate against the second one, and measuring accuracy in both. With each epoch I create new training/validation sets.
Witch each training run, I can see the training set accuracy increasing, reaching +98%. But validation set accuracy gets stuck at around 88% consistently.
So my question is, given the fact that both training and validation are randomly selected in each iteration from the same total samples (the 10.000 samples cited above), how can validation accuracy and training accuracies converge to different values?
At the end of the day, with each iteration, each new 8000/2000 pair would contain a large number of overlapping samples with sets of previous iterations, specially if taken out of only 10.000 total. I mean, it says that validation set has no repeated sets with the training set, but that's only for each training/validation creation. Why does validation behave differently, and not converge to training accuracy in the long run?