1
$\begingroup$

I have read very well the awesome answers and suggestions by @cbeleites and @Dikran Marsupial here for nested CV but I am still confused about something:

Basically now I understand that nested CV is not used for model selection but rather for estimating the general performance of a certain model.

Question 1

Having this said, what shall we do in order to tune hyper parameters if we cannot do that in nested CV ? because as also mentioned in the same link, we end up with K models (with K being the number of folds in the outer loop), and its not a good practice to choose the best out of those K models!

Shall we do regular cross validation again separately AFTER nested CV in order to tune hyper parameters ? (Although this is what the inner loop of the nested cv is about). I have read something similar in this thread

Question 2

If we want to report the general performance of this model with nested CV: Can the mean/std scores of all the folds of the outer loop resemble 'testing' performance and the mean/std scores of all the folds of the inner loop resemble 'validation'? I used to do this when using 'regular' (not nested) cross validation:

  1. split the data into 80% training and 20% testing
  2. do cross validation on the 80% training which internally does training/validation splits with hyper parameter tuning, and report 'validation' performance
  3. choose the best model that contributed to the best score in the cross validation and use it to train again on the whole training data, then test on the 20% testing and report 'testing' performance.

However, this 80/20 split procedure is done several times (specifically K times) in the nested CV strategy.

So to make a long story short, can I achieve this effect with nested CV (my main goal is to report validation and testing performances)?

Many thanks in advance.

$\endgroup$
1
$\begingroup$

Nested CV is used for both model selection (inner) and model evaluation (outer). Your steps 1-3 occur once in each outer fold. Only the inner loop can be used for model selection. The outer loop is used for evaluation. So the inner loop will give you what you call validation or “tuning performance”.

You should never tune after nested cross validation - you are correct that you must not choose one of the K models after the fact. Rather your estimate of performance from the outer folds corresponds to the model developed by applying the full modeling procedure (including tuning using cross validation) on the entire dataset.

$\endgroup$
  • $\begingroup$ Thank you @user0, my only concern is that as you have said, tuning is done in the inner loop, but not ONCE, but rather several times (the number of folds in the outer loop), and in each time we might get a different set of hyperparams. Having this said, which hyperparams set should we choose ?? $\endgroup$ – Hiyam Aug 25 at 8:46
1
$\begingroup$

The general workflow concerning nested cross validation is this:

  1. Perform nested cross validation, this gives you an estimate of your generalization error. As you have mentioned, this gives you K sets of hyper-parameters that may or may not be different. You do not simply want to pick one of these hyper-parameter sets or take the average or the mode or anything else, you will largely just disregard these (though it might be smart to check them for stability; if you get wildly different hyper-parameter sets across folds it may require some further research).

  2. Once you have your generalization error from step 1, you now perform cross validation on your entire dataset to tune your hyper-parameters. This is the set of hyper-parameters that will build your final model. Disregard the error rates you get during this cross-validation, they are not your error rate, the error rate from step 1 is your error rate. There is a common misconception that when you estimate an error rate you estimate it for a given set of hyper-parameters. Rather the error rate estimate is for the algorithm as a whole given your data not a specific instance of that algorithm if that makes sense.

With regards to your second question, yes you could consider the mean of the inner loops to be your "validation performance". Nested cross-validation is analogous to your "regular" CV procedure done many times with different folds and averaged.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.