I didn't find any similar question so I just post the question here.
Suppose after training and validation, the model performs poorly on the test set. Then what we do is to consider another model (Is it true? correct me if I'm wrong). When it still performs poor on the test set after training and validation, we head to another model. If we continue this process, does it result in overfitting on the test set in terms of model selection?
Thanks.
Yes this will result in overfitting to the train + "test" set.
Data-driven model selection is part of the training process: any kind of data used in order to obtain the final model is involved in training.
The fact that you select the model manually (instead of letting an optimization routing select a model) does not change this.
You'll find lots of questions and answers about this under the topics of (data-driven) model optimization and model selection.
Yes, if you will repeat this process multiple times you will overfit the test set. It happens in many Kaggel competitions when the group submit their results multiple times and try to tweak their models to produce better results on the public test set. Then when the competition ends and the private test set scores are calculated, their score drops dramatically. This is because their model overfits the public test set and can't generalize to new datasets.
A better approach is first split to train and test. Then derive your conclusion as much as you can on the train set. Means:
Perform nested-cross validation on the train set to hypertune parameters and to evaluate your models (inner loop - hyperparameter tunning, outer loop - model selection). Notice that the same model with different hyperparameters is considered as two different models for the model selection phase.
For the 1-3 best models: Analyze the difference in the results between the inner loop to the outer loop. Treat the outer loop scores as the test scores and the inner loop as the validation score.
*If the mean error of the inner loop is not satisfactory for you (compare it to the state-of-the-art or to human level), you have a bias problem. You'll want to build a more complex model (more trees, more layers in NN, train NN longer), feature engineering, feature selection.
*If the error std of the outer loop is high, or the mean error of the outer loop is higher than the inner loop means you have a variance problem. Your model is not stable and you may overfit to noise. You'll want to build a simpler model (i.e fewer trees in RF, simpler NN), add more data, regularization, check if splits are representative enough, etc.
It will not result in the over-fitting of the test set as the test set has not been used in training the model.
