1
$\begingroup$

Apologies if this has been answered before elsewhere. Answers I have read so far have only confused me further. Essentially, I want to check whether I can use the test set to choose betweeen two different models (say a SVR and a random forest regressor), after I have tuned their optimal parameters through cross-validation.

Here's my workflow:

  • I have divided my dataset into a training and a test set.
  • I use cross-validation with $k$-fold on the training set to select the model's best hyperparameters (i.e. those that will minimise the CV-error). This would be for example via a grid search to select the max_depth of a random forest regressor.
  • once the hyperparameters have been chosen, I fit the corresponding model on the whole training set.
  • I can then evaluate its performance on the test set.

Now I want to choose between the SVR and the random forest regressor.

  • Do I compare their performance on the test set and choose the one with the lowest error? In doing so, am I not contaminating the design of the model with knowledge of the test set?
  • If the above is not possible because the test set is supposed to be treated as unseen data, do I then choose the tuned model that had the lowest CV-error between the two? In that case, what's the point of having the test set at all and am I not wasting valuable data by setting it aside?

Thanks

Improve this question
$\endgroup$
2
  • $\begingroup$ You are correct, strictly speaking choosing based on test set performance is contaminating the design. What you can do is (if you have enough data) create a train-validation-test split and choose based on the performance with the validation set. If not, choose based on CV-error. The point of the test set is to estimate how your model error will behave when it faces data it has not seen, in this sense it is not a waste. Here is an interesting blogpost showing the importance of test sets blog.mrtz.org/2015/03/09/competition.html $\endgroup$
    – Jose Pliego
    Commented Sep 29, 2021 at 17:29
  • $\begingroup$ A train validation test sit is the same thing (but worse) as a train test split with cross validation done on the train set. Cross validation on the train king set is just repeated iterations of a train validation split $\endgroup$
    – astel
    Commented Sep 29, 2021 at 18:20

1 Answer 1

Reset to default
4
$\begingroup$

You are correct, choosing between two algorithms based on their test set evaluations causes contamination of your test set. What you should do is select the best performing algorithm during the CV stage and evaluate only that algorithm on the test set. Think about it this way, when you are choosing between a random forest with tree depth x vs tree depth y during CV this is no different than choosing between random forest with tree depth x and a SVR. Consider the algorithm selection as part of the hyper parameter tuning.

It is important to note also that the test set is never wasted data. After the generalization error is estimated using the test set, the appropriate action is to recombine your data again and perform CV on the whole dataset road select again your hyper parameters/algorithm. In this second iteration ignore the error rates you find as you already determined that in the previous step

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for your answer - it does clarify and reinforce my understanding. I have however a follow-up question: Say my first CV led me to a choice of hyperparameters $h_1$ with a related CV-error $p_1$. I then evaluate the model on the test set and I am dissatisfied with the result. I go back to my CV and for example extend my grid search on max_depth to a wider range. I now have a CV-error $p_2$ that is lower than $p_1$ and a test error also lower. Is it ok then to choose the set of hyperparameters $h_2$ even though I sort of used my knowledge of the test set to change my CV scheme? $\endgroup$
    – wissam124
    Commented Sep 30, 2021 at 13:28
  • $\begingroup$ I understand that of course what would be completely wrong is the case where, for example $p_2$ is higher than $p_1$ but the error on the test set is lower. In that case I am definitely using knowledge of the test set in the design of my model. $\endgroup$
    – wissam124
    Commented Sep 30, 2021 at 13:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.