The caret package is a brilliant R library for building multiple machine learning models, and has several functions for model building and evaluation. For parameter tuning and model training, the caret package offers ‘repeatedcv’ as one of the methods.
As a good practice, parameter tuning might be performed using nested K-fold cross validation which works as follows:
- Partition the training set into ‘K’ subsets
- In each iteration, take ‘K minus 1’ subsets for model training, and keep 1 subset (holdout set) for model testing.
- Further partition the ‘K minus 1’ training set into ‘K’ subsets, and iteratively use the new ‘K minus 1’ subset and the ‘validation set’ for parameter tuning (grid search). The best parameter identified in this step is used to test on the holdout set in step 2.
On the other hand, I assume, the repeated K-fold cross-validation might repeat the step 1 and 2 repetitively as many times we choose to find model variance.
However, going through the algorithm in the caret manual it looks like the ‘repeatedcv’ method might perform nested K-fold cross validation as well, in addition to repeating cross validation.
My questions are:
- Is my understating about the caret ‘repeatedcv’ method correct?
- If not, could you please give an example of using nested K-fold cross validation, with ‘repeatedcv’ method using the caret package?
Different cross validation strategies are explained and compared in this methodology article.
Krstajic D, Buturovic LJ, Leahy DE and Thomas S: Cross-validation pitfalls when selecting and assessing regression and classification models. Journal of Cheminformatics 2014 6(1):10. doi:10.1186/1758-2946-6-10
I am interested in “Algorithm 2: repeated stratified nested cross-validation” and “Algorithm 3: repeated grid-search cross-validation for variable selection and parameter tuning” using caret package.