I am training a deep neural network for classification (specifically, a convolutional neural network for object recognition). I use mini-batches for my training, because I cannot fit the entire training set onto my GPU. The total size of my training data is 10,000 images, and each minibatch is of size 100. For each training epoch, the training data is randomly shuffled, and then mini-batches are chosen one after the other until all training data has been used for that epoch. For each mini-batch, I record the training error for that mini-batch, before the mini-batch is processed, together with the testing error, which is computed over all 1,000 testing images.

What I am finding, is that during the first epoch, the training error for each mini-batch is around 10% lower than the testing error. As the first epoch progresses, both the training and testing error decrease, but generally with a 10% margin between the two. Of course, training error is typically lower than error, but this only makes sense when those same data have already been used in training. In my case, I am seeing this behaviour in the first epoch, such that when I compute the training error for a mini-batch before it is processed, the system has never seen this data before.

What could be the reason for this? I have tried different random shuffles of the data, and the same thing occurs. My only explanation is that the training data generally is easier to separate than the testing data, but I would be surprised if this is the case to such an extend, given that the training-testing split is random itself....

Any thoughts? Thanks!

  • $\begingroup$ Try to decrease the number of neurons and see if the margin tightens. $\endgroup$ – rcpinto Apr 7 '16 at 18:21
  • $\begingroup$ What is your error measure? MSE? Number of misclassified samples? $\endgroup$ – Luis Jul 3 '16 at 17:20

If your training and test data are drawn from the same distribution, shuffled etc, then training and test error for unseen minibatches, prior to training on those minibatches, should on average be identical.

If they are not, then one of those assumptions is being violated, ie:

  • your training and test data are not drawn from the same distributions, and/or
  • you didnt shuffle the data

Note that you have to be a bit careful about shuffling your data, its not always applicable, eg if you have time series: your test data should be drawn from a time period after the time periods from which all of the training data were drawn. But now the distributions are not the same (probably change gradually over time), which violates assumption 1, above, and would hence give rise to what you are seeing.


Not the answer you're looking for? Browse other questions tagged or ask your own question.