My problem: I have some rule based algorithms and various machine learning algorithms (random forest, boosting, ...) I want to compare for a specific use case.

Since I want to optimize the hyper parameters for my classifiers I think I need to split my (small) dataset in 3 partitions:

  1. Training data
  2. Validation data
  3. Test data

I perform hyper parameter tuning on the training data and validate on validation data. After I found the "best" parameters I train the model with the best parameters on my training data and test on my test data.

I could do this for all my algorithms (rule based won't need training, I'll just test them on the test set) and have in my opinion comparable results.

For parameter tuning I want to use GridSearchCV and/or RandomizedSearchCV which both validate using cross validation with the specified amount of folds. In this case I would not need the validation data set and purely use the training set for parameter tuning.

In the end I test again on my test set. And here is my problem - Is it really necessary? GridSearchCV for example will do cross validation for all permutations of parameters I set and come up with a mean accuracy or something else. Is the mean accuracy of this cross validation not a meaningful metric for future predictions? I don't see the point of testing again on the test set.

I'm a bit concerned because my dataset is relatively small (3000 datapoints on 12 classes with 14 features, imbalanced), so if I just test on the test set (without cross validation) there might be noise?

If someone has tips for a general approach I should pursue, thanks!


2 Answers 2


Yes, as annoying it can be, it is really important to test your best model on the test data.

It is the same thing as when you train on the training data: you won't validate on the same data. There, your hyperparameter tuning is part of the training, so that you won't test on the data you have used to train your hyper-parameters, namely training and validation data.

  • $\begingroup$ Alright, but is there anything I can do about my concern of only relying on 20% test data of my already small data set als basis for the comparison of all algorithms? $\endgroup$
    – Christian
    Apr 12, 2018 at 9:29
  • $\begingroup$ Ideally, you should select the best algorithm based only on the CV results. The test set is just to get more realistic figures on your real accuracy (or whatever other metric). $\endgroup$
    – Stergios
    Apr 13, 2018 at 9:46

If you do not use separate test, your result can be overfitted. In effect, the procedure choose the best model (among models from the same algorithm but with different hyperparameters) based on their validation loss, so there is not any guarantee that the model will generalize as well as the validation performance.

  • $\begingroup$ The problem is not overfitting (the cross-validation based model selection using the training set is used to solve that problem), but that the performance estimate will be optimistically biased. See jmlr.org/papers/volume11/cawley10a/cawley10a.pdf $\endgroup$ Dec 27, 2021 at 8:55
  • $\begingroup$ @Dikran Marsupial, yes, i mean our hyperparameters are selected based on validation and the measures are not reliable. $\endgroup$
    – keramat
    Dec 28, 2021 at 6:06
  • $\begingroup$ yes, that is my point, that is not the same thing as overfitting. Evaluating on a test set does not prevent overfitting, just allows you to detect over-fitting the validation set. Note that over-fitting the validation set can lead to over- or under- fitting of the training set. $\endgroup$ Dec 29, 2021 at 9:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.