What do you do after your tuned model perform badly on the test set? Suppose that I have splitted my dataset into training, validation and test set.
Now I have trained a network, and then performed a set of hyperparameter tuning on the validation set. I have reached a pretty good performance on the validation set.
Then finally you run it on the test set and it gave you a pretty large drop in accuracy.
What do you do next? Of course, you cannot tune your model further.
Do you re-run the model with entirely new initialization/splits/shuffled dataset and re-do the experiment? But then you have also learned a little bit from your test data from the previous experiment, which means you are biased in the next round when you re-train your model (for example, you are more likely to try out the same validation method as last time, maybe even use the same parameters). Is my observation correct?
 A: Generally speaking, this should not be the case, and is most likely an implementation bug. The validation performance should be very close to the test performance. If this is not the case, either:
A) [Most likely] the code has one of the following mistakes:

*

*Possibility 1: Incorrect preprocessing of the test set. E.g. applying some sort of preprocessing (zero meaning, normalizing, etc.) to the train and validation sets, but not the test set.


*Possibility 2: Testing the model in train mode. Certain layers such as batch normalization perform differently at training and inference time.


*Possibility 3: Some other implementation-related bug.
B) the validation set and test set come from very different distributions.
C) the dataset is small with an even smaller validation set.
A: You have overfitted the training set. Try again with more data, or with some form of regularization, possibly including added noise.
A: This may be due to your dev set and test set not being identically distributed.
One way to test this is to train a classifier that discriminates between the training/dev vs the test set.
If your dataset is small you should definitely check if the drop from dev/test metrics is consistent between splits. If the drop varies, you should do a nested cross validation. That way you average over the splits (which is random) and get a better estimate of the true performance.
