3
$\begingroup$

I refer to this extremely well explained post here. Link to the post

The confusion arises because after you have tuned your hyperparameters, and found the best performing classifier, be it a single model, or an convoluted ensemble, you found that that hypothesis $g$ performs best during cross-validation. The idea is then also use this $g$ to predict on X_test to see how well it does on the unseen data.

My question is, out of the all the models I trained, say $h_1, h_2, ...$ from various hypothesis space, or to put it simply, I trained a linear model, a tree model, and another ensemble model like Random Forest. Then eventually, I found the random forest to do best on the validation set, at last, I will use this model to predict on X_test to assess it. Why can't I use all 3 trained models to also perform on test set after the dust is settled...? Is it because it can introduce "my own bias" after I realized that maybe the ensembled model didn't get the highest score on X_test?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Any time you use a chunk of data to search through a possible space (whether that’s find best coefficients for a regression or choosing between RNN and CNN) you are by definition fitting to that data.

  • Training data: Used to fit a particular model.
  • Validation data: Used to search through the space of models to find best hyper parameters.
  • Testing data: Used to check whether your best model from Validation performs well OOS.

That’s the usual set up. However, as you said in your question you might want to take a couple of models from validation (say the top 5) and see which one generalises best on the testing.

As I said above, you will now be fitting to the test data and introducing the chance that you picked a model which got lucky. In this case we would create a fourth data set:

  • Holdout data: Kept completely separate and used for the sole purpose of assessing the OOS of a potential production model. Only one model should get to see this data.
Improve this answer
$\endgroup$
5
  • $\begingroup$ oh, good god. What if my dataset is like than say, 5000 rows (tabular), not image/deep learning. Then what is the tradeoff here? Is there a rule of thumb for me to use KFolds, or train-val-test split? $\endgroup$
    – nan
    Commented Sep 26, 2021 at 9:35
  • 2
    $\begingroup$ The sample size needed to get reliable/stable split sample validation is approximately n=30,000. Smaller sample sizes than that will yield estimates that are to subject to the randomness of the splits. For your case, programming all aspects of model development and repeating those steps afresh 1000 times will work much better (i.e., 100 repeats of 10-fold cross-validation). It is too expensive to hold back data from training if your sample size is not very large. (For very high signal:noise situations you can get by with smaller N). $\endgroup$
    – Frank Harrell
    Commented Sep 26, 2021 at 12:43
  • 1
    $\begingroup$ I agree, in my answer I’m describing the ideal situation. With small data, it’s important to be pragmatic and accept that there may be a chance of overfitting. The amount of data needed is highly problem dependent and will depend on the signal/noise ratio. $\endgroup$
    – Adam Kells
    Commented Sep 26, 2021 at 13:59
  • $\begingroup$ @FrankHarrell Dear Professor Frank, it's me again, are you referring to RepeatedKFold. So in your example, RepeatKFold 100 times, with different seeds? $\endgroup$
    – nan
    Commented Sep 27, 2021 at 9:37
  • 1
    $\begingroup$ You can pick one seed to start. The seed does not need updating because everything is random after that. Over 100 repetitions do ordinary 10-fold cross-validation to estimate performance. Average over the 100. In the inner loop repeat all supervised learning steps. $\endgroup$
    – Frank Harrell
    Commented Sep 27, 2021 at 11:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.