I am training 3 different models, with varying parameters like learning_rate, regularization_strength. But number of epochs is same for all. For a fixed no. of epoch, model 2 has the highest validation loss value. But the accuracy of it is highest in test set. How can it be possible ? Or I have some bug ?
$\begingroup$
$\endgroup$
4
-
1$\begingroup$ You should edit our question because it is unclear what you are asking. $\endgroup$– Michael R. ChernickCommented Mar 29, 2017 at 14:59
-
$\begingroup$ Are you using a cross-entropy loss function? $\endgroup$– itdxerCommented Mar 29, 2017 at 15:07
-
$\begingroup$ @MichaelChernick I rephrased the title. I hope it helps. $\endgroup$– ShyamkkhadkaCommented Mar 29, 2017 at 16:39
-
$\begingroup$ @itdxer, I think it has not anything to do with loss function. In anyway, I am using hinge loss function. $\endgroup$– ShyamkkhadkaCommented Mar 29, 2017 at 16:41
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
The accuracy of the learned model on the validation set is just a proxy for its accuracy in the real-world. It is possible that the accuracy of a learned model in the real-world (for example in the test set), be different than its estimated accuracy during training. If the difference between the accuracy of the model on the validation set and the test set is significant, it means that your validation set and test set comes from two different distributions. It is a common advice for machine learning practitioner that try to choose validation and test sets that are drawn from the same distribution. On the other hand, there is a topic in the machine learning called domain adaptation which tries to develop learning methods that perform well on a different target distribution.
$\begingroup$
$\endgroup$
It is possible to end up training a model which performs better on the test set than the training set, especially if appropriate regularization is used.
To be sure that you've trained a good model, you should try to get different training and test sets and see how the accuracy changes for the same hyper-parameters of the neural net. This will reveal if the "surprisingly good" performance of that model on the test set remains good for different combinations of training-test sets.
If this is not possible, and if you have a large enough number of training samples, then you could create multiple cross-validation folds and train-test on these folds. This should give you a good indication of how stable is the model on previously unseen data. This is the preferred method for training a model. See this wikipedia entry for more details and references.
answered Mar 30, 2017 at 4:28