I would like to make a predictive model to classify some test data set. So I'm thinking of utilizing several different models and find out the best one(by misclassification or other measures). I know that validation is an important step in the process to give us an accurate, out of sample performance. My question is, in the case with tuning parameters, when are they tuned in the validation process?

Say I use 10-fold cross validation, 100 times. Do I tune the parameters(again using cv) prior to the 100 10-folds, or tune it within each of the 100 or something else?


You'll want nested cross-validation. The short answer to your final question is you do it for each of the 10 folds, each of the 100 times you're repeating the 10-fold cross-validation.

Just pretend each split of the data is the only one you're using -- what would do then? Well do that for every split you're actually using.

The tuning of the hyperparameters is part of the model-building process, so it has to be done separately from the final training set (or hold-out samples), otherwise you'll be overestimating your performance as a result of using your validation data to train the model.

So to answer your question in a way that is hopefully generally applicable: whenever you want to test your method on any holdout set, perform the full model fitting procedure before you do so, and every time you do so.

You are pretending that the holdout set is data that doesn't exist... until the exact moment you want to see how your whole modelling procedure would perform given new data.

I think people get confused by doing this lots of times, but the only reason you do cross-validation many times is to smooth out the quirks of a particular dataset, you're trying to average over lots of possible new data sets to get a less biased picture. But the principle is the same as if you just had a straight up training/validation split -- you have to do it every time.

Practically, you have a for-loop inside a for-loop; the outer loop tests the model performance for each fold and the inner loop tunes the hyperparameter(s). You might then have another for-loop outside both of these that repeats the procedure 100 times.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.