4
$\begingroup$

I am having some doubts about understanding nested cross-validation. I'm conducting research with a small dataset and would like to get the nested cross-validation design correctly. My dataset is $50\times 212$, where the response variable is the age of the subject (rows: subjects, columns: features). I have 4 age groups, I'm also conducting sub-groups classification (for example, the groups are ABCD, and sub-classification tasks include A vs. D, B vs. D, C vs.D etc).

I'm using the outer loop to evaluate generalization error and the inner loop for model validation.

Problem description

I have selected initially two classifiers: polynomial and linear SVM. Outerloop uses stratified K-fold sampling to train-test sets, where K = 5. The value of K is selected in order to keep data distribution similar in training and test datasets. The inner loop uses leave-one-out cross-validation (LOOCV).

Feature selection and classifier hyperparameter tuning are conducted by using MATLAB's sequentialfs function by using a forward selection algorithm. The input function handle is custom created SVM functions for linear and polynomial functions. Both utilize Bayesian hyperparameter optimization.

Bayesian optimization fits training data of the validation round during each optimization iteration using the selected classifier. Each optimization iteration is tested against the validation dataset. Finally, the best feasible hyperparameters for the SVM model are selected and the corresponding loss is returned. I'm using hinge loss in this case.

The features are added to the model one by one until the selection criterion is no more satisfied. I'm using sequentialfs function's default tolerance value for criteria.

After each inner loop round, the best SVM classifier (linear or polynomial SVM) with its optimized hyperparameters is saved. After the inner loop is finished, the best-performing classifier is selected based on the smallest loss value.

Here is a point I don't understand: when using LOOCV in the inner loop, it's fairly likely that many models yield the same loss, as the validation set consists of one sample. How should I select a model in the case of ties in the inner loop?

After studying the nested cross-validation process, I understand that in essence, I'm validating the model selection process in the inner loop. Thus I won't select features but the hyperparameters for feature selection. After the inner loop, I conduct feature selection one more time with all the training data for the outer loop round using the selected classifier. Here I don't understand if should I select hyperparameters for the classifier in the inner loop or conduct this also after exiting the inner loop.

Summary of doubts

  1. How should I select a model in the case of ties in the inner loop (tie: loss is the same between two or more validation rounds)

  2. Should I lock the classifier hyperparameters based on the model selected based on inner loop results or should I conduct hyperparameter optimization one more time with the full training dataset after exiting the inner loop (together with feature selection)?

$\endgroup$

1 Answer 1

1
$\begingroup$

The best way of looking at it is that outer cross-validation is estimating the performance of a method of fitting a model. In that case, this method involves hyper-parameter tuning and feature selection based on leave-one-out cross-validation. So to produce the final model, just apply that method to the full dataset. I wrote a paper on this topic (with Mrs Marsupial) that I hope will be useful:

GC Cawley, NLC Talbot, On over-fitting in model selection and subsequent selection bias in performance evaluation, The Journal of Machine Learning Research 11, 2079-2107 (2010) (pdf)

$\endgroup$
2
  • $\begingroup$ Thank you! I read the paper and it gave great insights to the model selection process. I'm still bit unsure with my question 1. regarding LOOCV. I think my previous understanding of LOOCV model selection may have been faulty. I had understanding that the best performing model is selected based on the best single validation loss of the inner loop round. I think this could lead to poor performing model as it was stated that LOOCV introduces high variation in the results. Thus I see that averaging of the models performance is a better way to conduct model selection. $\endgroup$
    – masto12
    Dec 27, 2022 at 14:45
  • $\begingroup$ For example, if I use two classifiers (as in my original post) and the rest of the feature selection and classifier hyperparameter optimization hyperparameters are locked, I'll have at the end of the inner loop two competing models with computed validation errors across all the folds of the inner loop. And I select the model fitting method based on these results. Is this correct way to looking at use of LOOCV in model selection? $\endgroup$
    – masto12
    Dec 27, 2022 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.