1
$\begingroup$

I'm a little confused here. Appreciate if anyone can help me out.

During a k-fold nested CV, I understand for each combination of training fold and testing fold , the training fold will be further split into k subsets and a small CV will be carried out to determine the optimal hyper-parameters.

My question is: are the models generated by each training fold the same? If not, which model should be used when apply to productive environment.

I know there is a similar thread 'Nested cross validation for model selection'. But I still don't quite get it after reading for several times. Please bear with me as I just step into the field.

Thanks in advance.

[Note]

After reading some other threads, I can understand the mechanism below:

  1. Create hyper-parameter matrix;
  2. Use sub-training set to fit model;
  3. Use fitted model and validation set to select model;
  4. Use test fold to evaluate the performance of the chosen model.

However, my question is triggered by the book 'Data Science for Business'. When it describes 'Nested Cross Validation', it says:

... before building the model for each fold, we take the training set and first run an experiment: we run another entire cross-validation on just that training set to find the value of C estimated to give the best accuracy. The result of that experiment is used only to set the value of C to build the actual model for that fold of the cross-validation...

In my understanding, each fold may get a different value of C which leads to different models. Then which model should I use for productive operation?

Please help point out anything wrong in above understanding or if I have any misunderstanding of the book. Thanks a lot.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

The nested cross validation procedure above is used to find an unbiased estimate of the generalized performance as well as to determine the hyperparameter values.

For the final model, train using all available data and using the identified hyperparameters/features etc. Some good similar answers here . Hope this helps

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for answering. The reference is exact what I am looking for. So let's say I get K sets of hyperparameter α (which are the best sets for each fold) by inner loop. Then I use outer cross validation to examine the performance of each set of α and use the best one to create the model with the whole data set. Am I right? Thanks! $\endgroup$
    – Adrian.Lu
    Nov 20, 2013 at 6:07
  • $\begingroup$ Yes, hyperparameter selection and feature selection needs to happen on the inner-most cross-validation loop. The outer loop will give you a performance evaluation. Taking the chosen hyperparameters and features and applying to all data gives you your final model $\endgroup$
    – BGreene
    Nov 20, 2013 at 10:37
  • $\begingroup$ Thanks a lot. The answering and reference thread are really helpful. $\endgroup$
    – Adrian.Lu
    Nov 21, 2013 at 2:30
3
$\begingroup$

The following description of nested CV is my favorite so far and helped me a great deal to understand it (also here on Google Books): nested CV

(Petersohn, Temporal Video Segmentation, Vogt Verlag, 2010, p. 34)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.