8
$\begingroup$

In general, if we have a large dataset, we can split it into (1) training, (2) validation, and (3) test. We use validation to identify the best hyperparameters in cross validation (e.g., C in SVM) and then we train the model using the best hyperparameters with the training set and apply the trained model to the test to get the performance.

If we have a small dataset, we cannot create training and test set (not enough samples). Therefore, we will do cross validation (k-fold, leave-one-out, etc) to evaluate the model performance.

I have seen nested cross validation (whether repeated or stratified) has been used in the setting of small dataset, i.e., to generate generalized model performance while optimizing parameters selection. My question is, how can I obtain the best hyperparameters in nested cross validation (repeated/no repeated)? I am interested in doing this in scikit-learn, if possible. I am a bit confused about how to do it.

I have read several resources but none gave me the definitive answer for this question:

Nested cross validation for model selection

Nested cross-validation and feature selection: when to perform the feature selection?

$\endgroup$
  • $\begingroup$ This mentions scikit-learn, but has a viable machine learning question. This doesn't seem off topic to me. $\endgroup$ – gung Jan 5 '17 at 0:39
  • $\begingroup$ @gung yes, thanks. The scikit-learn is an additional part of the question ( a plus to me) $\endgroup$ – RockTheStar Jan 5 '17 at 0:42
6
$\begingroup$

Overview

As @RockTheStar correctly concluded in the commentaries, the nested cross-validation is used only to access the model performance estimate. Dissociated from that, to find the best hyperparameters we need to do a simple tuning with cross-validation on the whole data.


In details:

Tuning and validation (inner and outer resampling loops)

In the inner loop you perform hyperparameter tuning, models are trained in training data and validated on validation data. You find the optimal parameters and train your model on the whole inner loop data. Though it was trained to optimize performance on validation data the evaluation is biased.

So this model is tested with the test data so hopefully there's no bias, giving you a performance estimate.

The final model

Now that you know the expected performance of your model you have to train it with all your data. But our model isn't simply the algorithm, it's the whole model building process!

So perform hyperparameter tuning with all your data and the same specifications of the inner loop. With the best hyperparameters, train your final model with the whole data. The expected performance of this final model is what you evaluated with nested crossvalidation earlier.

To reiterate, the hyperparmeters of the final model is what you expect will give you the performance you found in the Tuning and validation step.

$\endgroup$
  • $\begingroup$ Thanks for your answer. I try to understand it but I am not sure if I get it. So where exactly I obtain my best hyperparameter set? $\endgroup$ – RockTheStar Jan 5 '17 at 0:44
  • $\begingroup$ @RockTheStar I updated the answer. Basically you refit (i.e. perform tuning again, and then fit the tuned model on all data): this is where your best bet for optimal hyperparameters come. $\endgroup$ – Firebug Jan 5 '17 at 1:41
  • 1
    $\begingroup$ so basically, you are saying the nested CV is to verify the model performance. And we need to do a simple CV for full data again to get the optimal parameters. Am I right? $\endgroup$ – RockTheStar Jan 5 '17 at 21:01
  • $\begingroup$ @RockTheStar Yes, you summarized it all correctly. $\endgroup$ – Firebug Jan 5 '17 at 21:05
  • 1
    $\begingroup$ @nafizh Sorry for taking so long to answer your inquiry, hadn't seen the notification (?). All performance estimates are to be based on the nested CV, which is the one you built to test your model building strategy. To perform the final hyperparameter tuning, you do the inner loop on the whole data and pick the best according to the same criterion you used in the Nested CV. Then you fit it to the whole data and that's your final model, but the predictive performance you ascribe to it is the one you obtained in the Nested CV. $\endgroup$ – Firebug Oct 17 '17 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.