I trained a resnet101 model using 900 samples / class (50 classes in total ), and chose the best model based on best accuracy using a separate validation set of 100 samples / class, throughout the whole training train/val accuracies were very close, which mean i am not over-fitting the train dataset(?), but the accuracy of a hold on test set of 100 samples / class was lower by 6~10% compared to validation accuracy, what could be the reason? NB: train/ val/test sets were sampled from the same distribution
$\begingroup$
$\endgroup$
4
-
$\begingroup$ possibly the test data happens to be different than train/val. Repeat the procedure with another random sampling to make sure that is not an issue. $\endgroup$– KrrrCommented Oct 25, 2017 at 7:41
-
1$\begingroup$ In general an estimate of the performance of the best-performing model on a validation set will be something of an over-estimate, the more so the more models you compare. That's why you have a test set: to get an unbiased estimate of a model's performance. $\endgroup$– Scortchi ♦Commented Oct 25, 2017 at 7:46
-
1$\begingroup$ What @Scortchi says. Problems arise if you use your results on the "test set" to modify your model - because then the "test set" effectively becomes part of the validation set, which I argue is in turn actually part of the training set. $\endgroup$– Stephan KolassaCommented Oct 25, 2017 at 7:48
-
$\begingroup$ just for clarification the val set was used along training at each epoch to monitor performance out of sample, best performing model is saved, while training continues for a chosen total epochs. @StephanKolassa the test set is only for testing generalization, no further modification is done to the selected model. $\endgroup$– chahrazadCommented Oct 25, 2017 at 8:53
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
You may be "overfitting to the validation set" at this step:
chose the best model based on best accuracy using a separate validation set
Once you tweak your model, or choose among multiple competing models based on the validation set, it is not a true validation set any more - it effectively becomes part of the training set, albeit in a somewhat obfuscated way.
Look at it in this way: suppose you train an astronomical number of models, including all your potential predictors, but also transformations, and completely unrelated predictors like sunspot numbers or frequencies of Mongolian throat singers. Then you apply all your models to the validation set and choose the one model that performs best on the validation set. This one happens to include an interaction term between last year's precipitation in Nowhere, OK and the hemlines at this year's Paris fashion shows - simply because you are fitting noise.
Would you expect this model to perform well on holdout test data, simply because it "worked" best on the validation set?
Unfortunately, there really is no good way around this dilemma. This can even happen with cross-validation and regularization or tree pruning. Differences between accuracy on different test and validation sets are one potential hint that something like this is going on, though. Your best approach would be to be parsimonious in your initial selection of candidate models, and collecting a lot of test data so you can use a "layered" approach with multiple iterated validation and test sets.
answered Oct 25, 2017 at 7:43