We have 3000 samples for two classes, roughly 2000:1000.

Our plan is to train a classifier on the samples but first to set aside 30% randomly selected stratified samples as a "holdout data set" for a final test.

Then we want to experiment a lot with the remaining 2100 samples, i.e. feature engineering and tweaking of classifier settings. Here we plan to use cross validation at each "tweak" of the classifier (measuring accuracies etc.). Then when we think we are done, run a final test on the 900 "holdout" samples with the final classifier.

Is this a valid use of the "holdout data set" and cross validation methods?

I have seen cross validation used before, but not with a separate testing data set as described.


1 Answer 1


What you did is perfectly valid. We always set aside a portion of data for testing what will happen when our model faces real life data. Training data as the name implies is the data segment where the model is trained while the validation is required for model selection and parameter tuning stages.

However, there is a better way called nested cross validation where data is split into training,validation and test segments repetitively with different combinations (https://stackoverflow.com/questions/42228735/scikit-learn-gridsearchcv-with-multiple-repetitions/42230764#42230764).

  • $\begingroup$ Is nested cross validation only for hyperparameter tuning? We are planning to add/remove features more than hyperparameter tuning (i.e. not easily automated). Also is the test sements (or fold) only to be used at the very end (i.e. only final evaluation)? If not wouldn't it affect how we train the model (inner CV)? $\endgroup$
    – Zoom
    Apr 20, 2019 at 20:13
  • $\begingroup$ You can try out any kind of processing steps within these folds including playing with features. For your second question, the test segments is not used at the very end, but in each step. As a result of this whole operation, you do not expect to obtain a ready-to-use model, instead you try to obtain the best model with its best parameters (including the things you did with features). Once you decide on these, then you can use all labeled data (including training, validation and testing) for a final training and use it with real data. $\endgroup$
    – Monotros
    Apr 21, 2019 at 17:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.