2
$\begingroup$

I have a quite big dataset of 10000 training data, i held out 2000 points for validation.

I am using a Convolutional Neural Network and using minibatch stochastic gradient descent to minimize the RMS error, with minibatch size of 50,i am training for 1000 epochs.

The training loss in the final epoch turns out to be 2.72 (averaging over the minibatch (size 50)).

However the validation loss comes out much lesser, 0.33 to be precise (averaging over the 2000 validation points).

How do i interpret these values, i am thinking that averaging over 2000 values compared to only 50 in the minibatch is playing a part.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Did you modify the probability of dropout between the validation and training? This might be an important factor to explain the disparity in loss values.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Yes! I also thought this was the issue initially. But both have the same. I am printing Training errors only for every hundredth epoch and in that epoch i am setting the keep probability = 1.0 and for validation also it is 1.0. $\endgroup$
    – Vikash Balasubramanian
    Commented Aug 19, 2016 at 13:55
  • $\begingroup$ Okay I'm not sure i understand perfectly what you are doing but from my experience never disable the dropout during the training even if it's 1 out of 100 it will mess up your network. Secondly i feel like you are confusing validation set and test set : validation set is used once every 100 epoch for example, to check the evolution of your model on a set who is not the training set. This allow you to perform early stoping of the training if necessary. Test set is what you test your model on when the training is complete. The repartition of the 3 is usually 50%/25%/25%. $\endgroup$
    – LoulouChameau
    Commented Aug 19, 2016 at 14:28
  • $\begingroup$ Sorry, what i meant was every 100 epochs i am evaluating the model both on training and validation data, just for understanding this is not some production code, i just wanted to see what's going on. I just couldn't understand why the loss in training data could be less than validation data, except for the fact that i am evaluating on a batch of 50 training points and 2000 validation points, but since i am taking mean, that shouldn't matter right. $\endgroup$
    – Vikash Balasubramanian
    Commented Aug 22, 2016 at 8:59
0
$\begingroup$

You should always check what happens if you reshuffle and do the train/validation split again in these situations.

Data points are inherently more/less difficult to classify and not shuffling appropriately will skew the loss over different splits.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.