Apologies if this has been answered before elsewhere. Answers I have read so far have only confused me further. Essentially, I want to check whether I can use the test set to choose betweeen two different models (say a SVR and a random forest regressor), after I have tuned their optimal parameters through cross-validation.
Here's my workflow:
- I have divided my dataset into a training and a test set.
- I use cross-validation with $k$-fold on the training set to select the model's best hyperparameters (i.e. those that will minimise the CV-error). This would be for example via a grid search to select the max_depth of a random forest regressor.
- once the hyperparameters have been chosen, I fit the corresponding model on the whole training set.
- I can then evaluate its performance on the test set.
Now I want to choose between the SVR and the random forest regressor.
- Do I compare their performance on the test set and choose the one with the lowest error? In doing so, am I not contaminating the design of the model with knowledge of the test set?
- If the above is not possible because the test set is supposed to be treated as unseen data, do I then choose the tuned model that had the lowest CV-error between the two? In that case, what's the point of having the test set at all and am I not wasting valuable data by setting it aside?