1
$\begingroup$

I trained a fully connected neural network with with five hidden layers of size $2024$ each. I used the Adam optimizer with a learning rate of $1e-4$ and a drop out rate of $0.4$. Batch size was $1000$. After about $24h$ of training I saw that the test accuracy is much higher than the training accuracy. How is that possible and how can I interpret this result?

enter image description here enter image description here enter image description here

EDIT:

I trained the network for the MNIST data set and rescaled test and training data by $2.0*(IMAGES/255.0-0.5)$ Therefore I assume, that both training and test set are equally distributed.

$\endgroup$
  • $\begingroup$ Your train and test data are most likely not identically distributed. $\endgroup$ – TenaliRaman May 11 '18 at 11:07
  • $\begingroup$ @TenaliRaman I edited my question. I hope that answers your concern. $\endgroup$ – I_told_you_so May 11 '18 at 11:41
  • 1
    $\begingroup$ You're maybe the only one in the world still using fully connected NNs for MNIST. Any specific reason why you did that? Also, your NN is hugely overparametrized for MNIST: I get much better results with way less parameters using a beefed-up LeNet. Are you sure you aren't training on the test set, and validating on the training set? $\endgroup$ – DeltaIV May 12 '18 at 9:06
  • $\begingroup$ I think this question can't be answered write looking at the code, and once you add code, it's no more a good fit for CV (it will be a good fit for SO, though). $\endgroup$ – DeltaIV May 12 '18 at 9:07
  • $\begingroup$ @DeltaIV I was just playing around with some implementations and did this test run to see if the results are plausible. $\endgroup$ – I_told_you_so May 12 '18 at 12:52
1
$\begingroup$

You need to ensure that the training and test sets are picked completely at random. If there is some selection bias introduced in this procedure, such that the separation into training and test patterns does not occur at random, you can end up in the situation where your training set contains relatively more 'difficult' patterns. With difficult I mean closer to the decision boundaries. In such a situation your performance on the training set may be exceeded by the test-set performance.

Shuffling your complete data set again, and performing a new separation into training and test sets should remedy the counter intuitive finding you report in your question.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I used the MNIST data set. Test and training set are completely disjunct. $\endgroup$ – I_told_you_so May 11 '18 at 11:43
  • 4
    $\begingroup$ then I believe your training accuracy is quite low for the MNIST dataset, and I would check for mistakes/bugs in its computation. $\endgroup$ – fabiob May 11 '18 at 11:53
  • 1
    $\begingroup$ What does disjunct mean to you? If it means completely separate that is not the same as random. Also disjunct is not a real English word. I assumed you meant disjoint. $\endgroup$ – Michael R. Chernick May 11 '18 at 13:33
  • 1
    $\begingroup$ @MichaelChernick he means that for MNIST the training set and the test set are defined in advance and disjoint: the training set contains 60000 digits and the test set contains 10000 digits. It's a well-known data set for handwritten digit classification. See yann.lecun.com/exdb/mnist $\endgroup$ – DeltaIV May 12 '18 at 9:10
  • 1
    $\begingroup$ @DeltaIV "This is most likely due to the architecture (or implementation bugs)" I totally agree, while 70% is without any shade of a doubt a bug/mistake, no food for thought about whether he separated randomly enough the train and test set (which as you pointed out, he did not, it is already done for MNIST). $\endgroup$ – fabiob May 12 '18 at 12:09
0
$\begingroup$

Scaling the dataset feature values doesn't mean that your train and test set is equally distributed.


Consider the following example:

I survey a group of 100 people - 80 Software Engineers and 20 Engineering Managers. I ask for all kinds of information like their hardware preferences, software preferences, income bracket they fall in etc. Finally, I ask them whether they will buy my product or not.

Now, I build a classifier with this data to predict whether someone from engineering world would buy my product or not. I find that my classifier is absolutely accurate for Software Engineers but it gets all Engineering Managers wrong. So, the training accuracy of my classifier is 80%

Now, I also have a test set where I had surveyed 100 more people - 50 Software Engineers and 50 Engineering Managers. I observed that my classifier's accuracy is just 50% on test set! Why?

It is not easy to figure this out because, in my training data, the classifier looked mostly at Software Engineers and started tuning itself accurately for their preferences. However, the distribution of the Software Engineers to Engineering Managers was quite different in train and test, thus we get a different accuracy number. This is what we call as distribution mismatch between train and test set. This happened because I was not careful while doing my survey. While doing my survey, I should have ensured that the process of selecting people for survey stayed the same while creating the train and test set. Clearly, something changed when I was selecting people, which is why we saw this difference in distribution. This is what we call as selection bias.


To avoid such issues, it is imperative that your train and test set are sampled uniformly randomly from the dataset so that you don't have any bias in the data while evaluating.

| cite | improve this answer | |
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.