# Repeated CV evaluation with confidence intervals in R caret?

it occurs to me that there is a part of model evaluation that I have not understood yet. The problem that I am working on now illustrates the point well I think.

I need to fit a model of >400 predictors to an ordinal response variable. The predictors are correlated of course, and penalized ordinal regression (LASSO) works reasonably well to build the model and reduce the predictors. Using caret in R I split the data into a training and testing data set, and tune parameters using cross-validation in the training data and I find a good model. I may even repeat cross validation multiple times "repeatedcv" to be assured that the model I built was just not odd because of unfortunate k-fold cross validation split.

Some illustrative code:

notna <- na.omit(fullData)

trainIndex <- createDataPartition(notna\$RatingCategory, p = .8,
list = FALSE,
times = 1)

training <- notna[ trainIndex[,1],] %>%
select(RatingCategory,FCoM_envel:ATrPS_freq,Jitter->F0_abs_dif:RPDE)
testing  <- notna[-trainIndex[,1],] %>%
select(RatingCategory,FCoM_envel:ATrPS_freq,Jitter->F0_abs_dif:RPDE)

fitControl <- trainControl(
method = "repeatedcv",
number = 10,
allowParallel=TRUE,
repeats=100,
savePredictions="final")

ordCVFit <- train(RatingCategory ~ ., data = training,
method = "ordinalNet",
trControl = fitControl,
preProcess=c("center", "scale"),
metric="Kappa",