1
$\begingroup$

Say you want to train the hyperparameters of an RBF kernel in a regression context but you have a limited number of data. You do 5-fold cross-validation. You split your data into 5 different sets and you use 4 of them as the training set and one of them as the validation set. You do this 5 times. You report several metrics on each validation set such that the RMSE. In this way, you obtain 5 different values of hyperparameters of your RBF kernel. Now, assume that you have some more data coming in and you want to test the different values of hyperparameters you have. Which one do you select? The set of hyperparameters that gave you the lowest RMSE? Moreover, to test your model on the test set, you train your model on the entire data set excluding the test set, right?

$\endgroup$
8
  • 2
    $\begingroup$ I would describe the process differently. For each potential value of the hyperparameters you are considering, you combine the results on the validation sets from the five folds, using the metric you care about. From this, you choose the hyperparameters you want to use in your final model. You then train your final model using these hyperparameters on all the data except for the test set. You now have your final model. You test your final model on your test set as a measure of how well it performs on unseen data. Then you stop (unless new data comes in to allow a new unseen test set). $\endgroup$
    – Henry
    Sep 17, 2021 at 12:50
  • $\begingroup$ Wait, so, how do you optimize your hyperparamaters? If you use gradient-based optimization, do you consider for optimization the sum of the metric of interest, for instance? $\endgroup$
    – Akusa
    Sep 17, 2021 at 13:29
  • 1
    $\begingroup$ That would depend on your particular model. The last time I did this, I used a grid-based search $\endgroup$
    – Henry
    Sep 17, 2021 at 13:37
  • 1
    $\begingroup$ Yes - so it is common to put some of the original data into a separate test set before you start any pre-processing or training and cross-validation stages, as a proxy for unseen data $\endgroup$
    – Henry
    Sep 17, 2021 at 13:50
  • 1
    $\begingroup$ That is what I was trying to describe earlier perhaps with the insertion "... 80% data, train the model with those hyperparameters and that data, then you test it ..." $\endgroup$
    – Henry
    Sep 17, 2021 at 13:59

1 Answer 1

2
$\begingroup$

If you do 5-fold-CV, you get 5 different estimates of performance (in terms of your selected metric) - one per fold - for the same hyperparameters. I.e. you train your model 5 times with the same hyperparameters on 4 of the 5 splits of your data, each time leaving out one split of the data as validation. Then you calculate the metric of interest on the bit of data you left out.

If you then do the same thing again for different hyperparameters, you get another set of performance metrics. It can be useful to look at the mean metric value, but also the standard deviation of metrics across splits. There's different approaches to then pick the hyperparameters to use later. The "greedy" approach of picking the hyperparameters with the best mean metric value (or best mean + SD in case you want to favor models with less variable outcomes) may overfit the validation a bit, so there's various other ideas like the "1-standard-error rule (see e.g. Chapter 7 of "The elements of Statistical Learning" by Hastie, Tibshirani and Friedman, 2009). The 1-SE-rule says to pick the hyperparameters that lead to the most penalization/regularization (with multiple regularization parameters such an ordering is not always clear), but which still lie within 1 SE of the best mean metric value.

Other ideas could include model averaging or other ensembling techniques like stacking, which try to reflect the uncertainty around model choice by making use of all models to the extent that it seems helpful.

Retraining on all non-test data is indeed a sensible option. However, it can occasionally be difficult, e.g. if you need a validation set to determine when to stop training and the training process is too random so that you cannot just use the training length that worked during cross-validation. Other options include averaging the predictions of the models trained on the 5 different training data sets.

$\endgroup$
7
  • $\begingroup$ So, we use a 5-fold CV. Denote by 1 2 3 4 5 each split set - First fold: You use (1,2,3,4) as the training set and (5) as the validation set. You optimize your hyperparameters using the training set and compute the metric of interest on the validation set to assess the performance of your hyperparameters. - Second fold: You use (2, 3, 4, 5) as the training set and (1) as the validation. You proceed the same as the first fold but you obtain different values of hyperparameters. - Third fold: same - Fourth fold: same - Fifth fold: same $\endgroup$
    – Akusa
    Sep 17, 2021 at 13:25
  • $\begingroup$ Thus, in total, you have 5 different sets of values of hyperparametes, one for each fold. Now, you have some unseen data that were not used for the 5-fold CV and you want to see if your model performs well. Which values of hyperparameters should I use? Naturally, I would take the hyperparameters for which the metric of interest on the validation set was minima but it might not be the best choice. Now, you use the hyperparameters selected and use them into your model with the entire data excluding the unseen data without re-optimizing your hyperparameters. Is that correct? $\endgroup$
    – Akusa
    Sep 17, 2021 at 13:25
  • $\begingroup$ No, you try one set of hyperparameters (HPs) & fit the model with those HPs for all 5 folds. The model parameters differ from fold to fold, but the HPs (e.g. ridge penalty in ridge regression) are the same for all folds. That evaluates the performance of training with those HPs. Then you repeat this with the next candidate HPs (however you choose those, e.g. grid search, random search, some kind of Bayesian model etc.). Repeat this a number of times (e.g. 1000). Then, you pick by some rule (e.g. the ones described in my answer) the best HPs and re-train on all data excl. the unseen test data. $\endgroup$
    – Björn
    Sep 17, 2021 at 18:34
  • $\begingroup$ Thank you. So, say you want to find your hyperparameters by maximazing the marginal likelihood estimation. On each fold, you compute the marginal likelihood using the learning set. You average them and compute their gradient with respect to the hyperparameters. You move the hyperparameters to the direction of gradient descent. To check at the same time if you are not ovetfitting, you can compute the RMSE on the validation set of each fold and take the average. After you are done, you get the values of your hyperparameters, you use them for your model with your entire data on some unseen data. $\endgroup$
    – Akusa
    Sep 17, 2021 at 21:24
  • $\begingroup$ Is that something one can do? $\endgroup$
    – Akusa
    Sep 17, 2021 at 21:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.