Test accuracy much higher than training accuracy I trained a fully connected neural network with with five hidden layers of size $2024$ each. I used the Adam optimizer with a learning rate of $1e-4$ and a drop out rate of $0.4$. Batch size was $1000$. After about $24h$ of training I saw that the test accuracy is much higher than the training accuracy. How is that possible and how can I interpret this result?



EDIT:
I trained the network for the MNIST data set and rescaled test and training data by $2.0*(IMAGES/255.0-0.5)$ Therefore I assume, that both training and test set are equally distributed.
 A: You need to ensure that the training and test sets are picked completely at random. If there is some selection bias introduced in this procedure, such that the separation into training and test patterns does not occur at random, you can end up in the situation where your training set contains relatively more 'difficult' patterns. With difficult I mean closer to the decision boundaries. In such a situation your performance on the training set may be exceeded by the test-set performance.
Shuffling your complete data set again, and performing a new separation into training and test sets should remedy the counter intuitive finding you report in your question.
A: Scaling the dataset feature values doesn't mean that your train and test set is equally distributed.

Consider the following example:
I survey a group of 100 people - 80 Software Engineers and 20 Engineering Managers. I ask for all kinds of information like their hardware preferences, software preferences, income bracket they fall in etc. Finally, I ask them whether they will buy my product or not.
Now, I build a classifier with this data to predict whether someone from engineering world would buy my product or not. I find that my classifier is absolutely accurate for Software Engineers but it gets all Engineering Managers wrong. So, the training accuracy of my classifier is 80%
Now, I also have a test set where I had surveyed 100 more people - 50 Software Engineers and 50 Engineering Managers. I observed that my classifier's accuracy is just 50% on test set! Why?
It is not easy to figure this out because, in my training data, the classifier looked mostly at Software Engineers and started tuning itself accurately for their preferences. However, the distribution of the Software Engineers to Engineering Managers was quite different in train and test, thus we get a different accuracy number. This is what we call as distribution mismatch between train and test set. This happened because I was not careful while doing my survey. While doing my survey, I should have ensured that the process of selecting people for survey stayed the same while creating the train and test set. Clearly, something changed when I was selecting people, which is why we saw this difference in distribution. This is what we call as selection bias.

To avoid such issues, it is imperative that your train and test set are sampled uniformly randomly from the dataset so that you don't have any bias in the data while evaluating.
