How to obtain optimal hyperparameters after nested cross validation?

Question

In general, if we have a large dataset, we can split it into (1) training, (2) validation, and (3) test. We use validation to identify the best hyperparameters in cross validation (e.g., C in SVM) and then we train the model using the best hyperparameters with the training set and apply the trained model to the test to get the performance.

If we have a small dataset, we cannot create training and test set (not enough samples). Therefore, we will do cross validation (k-fold, leave-one-out, etc) to evaluate the model performance.

I have seen nested cross validation (whether repeated or stratified) has been used in the setting of small dataset, i.e., to generate generalized model performance while optimizing parameters selection. My question is, how can I obtain the best hyperparameters in nested cross validation (repeated/no repeated)? I am interested in doing this in scikit-learn, if possible. I am a bit confused about how to do it.

I have read several resources but none gave me the definitive answer for this question:

Nested cross validation for model selection

Nested cross-validation and feature selection: when to perform the feature selection?

Answer 1

Overview

As @RockTheStar correctly concluded in the commentaries, the nested cross-validation is used only to access the model performance estimate. Dissociated from that, to find the best hyperparameters we need to do a simple tuning with cross-validation on the whole data.

---

In details:

Tuning and validation (inner and outer resampling loops)

In the inner loop you perform hyperparameter tuning, models are trained in training data and validated on validation data. You find the optimal parameters and train your model on the whole inner loop data. Though it was trained to optimize performance on validation data the evaluation is biased.

So this model is tested with the test data so hopefully there's no bias, giving you a performance estimate.

The final model

Now that you know the expected performance of your model you have to train it with all your data. But our model isn't simply the algorithm, it's the whole model building process!

So perform hyperparameter tuning with all your data and the same specifications of the inner loop. With the best hyperparameters, train your final model with the whole data. The expected performance of this final model is what you evaluated with nested crossvalidation earlier.

To reiterate, the hyperparmeters of the final model is what you expect will give you the performance you found in the Tuning and validation step.