Firstly, apologies for posting a question that has already been discussed at length here, here, here, here, here, and for reheating an old topic. I know @DikranMarsupial has written about this topic at length in posts and journal papers, but I'm still confused, and judging from the number of similar posts here, it's still something that others struggle to grasp. I should also state that I've received contradictory on this topic which has added to my confusion. You should also know that I'm originally a physicist and not a statistician, so my domain expertise here is somewhat limited. I'm writing a journal paper in which I want to use nested CV to estimate the performance I can expect from my final model. In my domain, this is a first. (We almost never use any form of robust CV in my field but merrily pump out papers with results from studies using neural nets and boosted decision trees!) Therefore, it's very important that I have a very thorough and clear understanding so that I don't screw-up and propagate an erroneous procedure to my community that could years to unlearn! Thanks! On with the question...
How do I build the final model after nested cross-validation?
I'm training a simple glmnet model with L1 and L2 regularisation. It's fast, simple and interpretable. I perform feature centring, scaling, and Box-Cox transformations to that the feature distributions are mean-centred, standardised and are somewhat Gaussian-like. I perform this step within cross-validation, to prevent information leakage. Purely because my hardware is incredibly slow and I don't have access to more CPU muscle, I also perform fast filter-based feature selection within CV after feature preprocessing. I'm using random grid search to pick the alpha and lambda hyperparameters. I understand that I shouldn't report the performance from the CV used to pick the optimal hyperparameters as an estimate of the expected performance of my final model (which would be overly-optimistic) but should instead include an outer CV loop to get this estimate. I understand that the inner CV loop is used for model selection (in this case, the optimal hyperparameters) and that the outer loop is used for model evaluation, i.e., the inner and outer CV serve two different purposes that often are erroneously conflated. (How am I doing so far?)
Now, the links I've posted suggest that "the way to think of cross-validation is as estimating the performance obtained using a method for building a model, rather than for estimating the performance of a model". Given that, how should I interpret the results of the nested CV procedure?
The advice I've read seems to indicate the following --- please correct me if this is wrong: the inner CV is part of the mechanism that allows me to select the optimal alpha and lambda hyperparameters of my glmnet model. The outer CV tells the estimate I can expect to get from the final model if I apply the procedure exactly as used in the inner CV including hyperparameter tuning and using the entire dataset to build the final model. That is, the hyperparameter tuning is part of "the method for building the model". Is this correct or not? Because this is what confuses me. Elsewhere I've seen that the procedure for building the final model to be deployed involves training on the entire dataset using the fixed values of the hyperparameters that have been chosen using CV. Here, the "method for building the model" does not include tuning. So, which is it? At some point the optimal hyperparameters are chosen and fixed for building the final model! Where? How? If my inner loop is 5-fold CV, and my outer loop is 5-fold CV, and I select, say, 100 points for testing as part of random grid search in the inner CV, how many times do I actually train the glmnet model? (100 * 5 * 5) + 1 for the final build, or are there more steps that I'm unaware of?
Basically, I need a very clear description of how to interpret the performance estimate from nested CV and how to build the final model.
I also would like to know the appropriate procedure for selecting the probability threshold for converting the probability scores from my final glmnet model into (binary) class labels --- another loop of CV is needed?