7
$\begingroup$

Suppose that I have splitted my dataset into training, validation and test set.

Now I have trained a network, and then performed a set of hyperparameter tuning on the validation set. I have reached a pretty good performance on the validation set.

Then finally you run it on the test set and it gave you a pretty large drop in accuracy.

What do you do next? Of course, you cannot tune your model further.

Do you re-run the model with entirely new initialization/splits/shuffled dataset and re-do the experiment? But then you have also learned a little bit from your test data from the previous experiment, which means you are biased in the next round when you re-train your model (for example, you are more likely to try out the same validation method as last time, maybe even use the same parameters). Is my observation correct?

$\endgroup$
3
  • $\begingroup$ Do you literally mean "accuracy" of a classifier (percent of correct classifications) or more colloquially in terms of any kind of measure of performance (e.g. crossentropy loss or mean absolute error)? Accuracy is a surprisingly poor measure of performance, strange as that sounds. $\endgroup$
    – Dave
    Oct 1, 2020 at 15:20
  • $\begingroup$ @Dave Well you can assume other type of measurements, but my point remains, suppose it performs bad on this measurement, what is the reasonable thing to do next given that you have already got a sense of what doesn't work. $\endgroup$
    – Olórin
    Oct 1, 2020 at 15:32
  • $\begingroup$ Have you tried cross validation with the original trainings + validation data? Maybe the drop in accuracy is well within the expected variance of the model. $\endgroup$ Jan 29, 2021 at 19:21

3 Answers 3

3
$\begingroup$

Generally speaking, this should not be the case, and is most likely an implementation bug. The validation performance should be very close to the test performance. If this is not the case, either:

A) [Most likely] the code has one of the following mistakes:

  • Possibility 1: Incorrect preprocessing of the test set. E.g. applying some sort of preprocessing (zero meaning, normalizing, etc.) to the train and validation sets, but not the test set.

  • Possibility 2: Testing the model in train mode. Certain layers such as batch normalization perform differently at training and inference time.

  • Possibility 3: Some other implementation-related bug.

B) the validation set and test set come from very different distributions.

C) the dataset is small with an even smaller validation set.

$\endgroup$
2
  • 3
    $\begingroup$ There's another possibility: the model selection procedure is overfitting the validation set. The risk of this increases as the size of the validation set shrinks and/or the space of models you search over becomes richer (e.g. many hyperparameters). Nonstationarity, covariate shift, etc. (option B) and high variance estimates of the generalization performance (option C) are also very real possibilities. So, I don't think it can generally be said that coding errors are the most likely cause. $\endgroup$
    – user20160
    Feb 19, 2021 at 13:36
  • $\begingroup$ I agree with @user20160 that it can be an overfitting to validation issue. Aside from that, this answer would benefit from answering the question "what do you do next" for each of the cases, possibly also including a diagnostic test for finding which is the culprit. $\endgroup$ Feb 21, 2021 at 14:42
0
$\begingroup$

You have overfitted the training set. Try again with more data, or with some form of regularization, possibly including added noise.

$\endgroup$
2
  • 7
    $\begingroup$ The trouble with this is that, by the time the OP has realized the overfitting, there are no more data left for a true test set! $\endgroup$
    – Dave
    Oct 1, 2020 at 15:47
  • 1
    $\begingroup$ @Dave raises an excellent point, which is why the bootstrap may be more favourable $\endgroup$ Oct 1, 2020 at 16:00
0
$\begingroup$

This may be due to your dev set and test set not being identically distributed.

One way to test this is to train a classifier that discriminates between the training/dev vs the test set.

If your dataset is small you should definitely check if the drop from dev/test metrics is consistent between splits. If the drop varies, you should do a nested cross validation. That way you average over the splits (which is random) and get a better estimate of the true performance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.