0
$\begingroup$

I am new on predictive modeling and for me it is not clear if nested cross validation is applied to full data set (all data I have) or just to traning data. For exemple, if it is applied to full data set, the outer CV is related to testing/training data and the inner CV to analysis/assessment data into the training data set?

Someone could help me with some idea? I am using caret package.

$\endgroup$
2
$\begingroup$

The idea is that you want to evaluate the predictive accuracy of your entire fitting procedure against holdout data. Nested cross validation is the application of CV inside of a CV training fold, i.e. yes, you apply it to your training data.

Here's a concrete example:

Let's say you are fitting a penalized model like lasso or ridge, and you want to use 10-fold cross validation to determine the appropriate regularization parameter. But, you also want to use 3-fold cross validation to evaluate your model's accuracy against heldout data. Just like fitting the rest of your model's coefficients, fitting the regularization term with 10-fold CV is now part of the fitting process, so the entire fit needs to occur inside each of the three cross validation folds to evaluate accuracy.

In other words (continuing the example), your first step will be to split the data into 67/33 train/test sets. Then, you split the training set 90/10 (so you'll now have the data divided as {60/7}/33)to fit a model to each candidate regularization parameter, which you evaluate on the other 10% (which is a segment of the 67% training data). Repeat this to complete the 10-fold CV (the "inner" nested CV), and you'll have enough information to pick an appropriate regularization term. Then, train the model (with that regularization parameter) against the full 66% training data, and evaluate against the 33% holdout set. Congratulations, you've just completed the first of the three "outer" CV folds.

$\endgroup$
0
$\begingroup$

Cross-validation (CV) is applied to the training data set only, when the goal is to use the developed model to make predictions on the test data set. The whole idea is to make the modeling decision and predictive assessment on different data sets.

In a nutshell, the CV score is calculated for many candidate models on the training data. Then the model with the lowest CV score is chosen. We commit to that model. To see if we are going to get ruined with our model selection process, or if we are going to become millionaires, we run the new model on the test data and calculate various performance metrics.

In practice, the idea can be applied to identifying and exploiting several most promising candidate models. In practice cross-validation must be combined with other goodness-of-fit diagnostics, whenever possible.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.