I am training a deep neural network for classification (specifically, a convolutional neural network for object recognition). I use mini-batches for my training, because I cannot fit the entire training set onto my GPU. The total size of my training data is 10,000 images, and each minibatch is of size 100. For each training epoch, the training data is randomly shuffled, and then mini-batches are chosen one after the other until all training data has been used for that epoch. For each mini-batch, I record the training error for that mini-batch, before the mini-batch is processed, together with the testing error, which is computed over all 1,000 testing images.
What I am finding, is that during the first epoch, the training error for each mini-batch is around 10% lower than the testing error. As the first epoch progresses, both the training and testing error decrease, but generally with a 10% margin between the two. Of course, training error is typically lower than error, but this only makes sense when those same data have already been used in training. In my case, I am seeing this behaviour in the first epoch, such that when I compute the training error for a mini-batch before it is processed, the system has never seen this data before.
What could be the reason for this? I have tried different random shuffles of the data, and the same thing occurs. My only explanation is that the training data generally is easier to separate than the testing data, but I would be surprised if this is the case to such an extend, given that the training-testing split is random itself....
Any thoughts? Thanks!
