How to select the best performing model when using nested cross-validation?

Question

I am having some doubts about understanding nested cross-validation. I'm conducting research with a small dataset and would like to get the nested cross-validation design correctly. My dataset is $50\times 212$, where the response variable is the age of the subject (rows: subjects, columns: features). I have 4 age groups, I'm also conducting sub-groups classification (for example, the groups are ABCD, and sub-classification tasks include A vs. D, B vs. D, C vs.D etc).

I'm using the outer loop to evaluate generalization error and the inner loop for model validation.

Problem description

I have selected initially two classifiers: polynomial and linear SVM. Outerloop uses stratified K-fold sampling to train-test sets, where K = 5. The value of K is selected in order to keep data distribution similar in training and test datasets. The inner loop uses leave-one-out cross-validation (LOOCV).

Feature selection and classifier hyperparameter tuning are conducted by using MATLAB's sequentialfs function by using a forward selection algorithm. The input function handle is custom created SVM functions for linear and polynomial functions. Both utilize Bayesian hyperparameter optimization.

Bayesian optimization fits training data of the validation round during each optimization iteration using the selected classifier. Each optimization iteration is tested against the validation dataset. Finally, the best feasible hyperparameters for the SVM model are selected and the corresponding loss is returned. I'm using hinge loss in this case.

The features are added to the model one by one until the selection criterion is no more satisfied. I'm using sequentialfs function's default tolerance value for criteria.

After each inner loop round, the best SVM classifier (linear or polynomial SVM) with its optimized hyperparameters is saved. After the inner loop is finished, the best-performing classifier is selected based on the smallest loss value.

Here is a point I don't understand: when using LOOCV in the inner loop, it's fairly likely that many models yield the same loss, as the validation set consists of one sample. How should I select a model in the case of ties in the inner loop?

After studying the nested cross-validation process, I understand that in essence, I'm validating the model selection process in the inner loop. Thus I won't select features but the hyperparameters for feature selection. After the inner loop, I conduct feature selection one more time with all the training data for the outer loop round using the selected classifier. Here I don't understand if should I select hyperparameters for the classifier in the inner loop or conduct this also after exiting the inner loop.

Summary of doubts

1. How should I select a model in the case of ties in the inner loop (tie: loss is the same between two or more validation rounds)

2. Should I lock the classifier hyperparameters based on the model selected based on inner loop results or should I conduct hyperparameter optimization one more time with the full training dataset after exiting the inner loop (together with feature selection)?

Answer 1

The best way of looking at it is that outer cross-validation is estimating the performance of a method of fitting a model. In that case, this method involves hyper-parameter tuning and feature selection based on leave-one-out cross-validation. So to produce the final model, just apply that method to the full dataset. I wrote a paper on this topic (with Mrs Marsupial) that I hope will be useful:

GC Cawley, NLC Talbot, On over-fitting in model selection and subsequent selection bias in performance evaluation, The Journal of Machine Learning Research 11, 2079-2107 (2010) (pdf)