it occurs to me that there is a part of model evaluation that I have not understood yet. The problem that I am working on now illustrates the point well I think.
I need to fit a model of >400 predictors to an ordinal response variable. The predictors are correlated of course, and penalized ordinal regression (LASSO) works reasonably well to build the model and reduce the predictors. Using
caret in R I split the data into a training and testing data set, and tune parameters using cross-validation in the training data and I find a good model. I may even repeat cross validation multiple times "repeatedcv" to be assured that the model I built was just not odd because of unfortunate k-fold cross validation split.
Some illustrative code:
notna <- na.omit(fullData) trainIndex <- createDataPartition(notna$RatingCategory, p = .8, list = FALSE, times = 1) training <- notna[ trainIndex[,1],] %>% select(RatingCategory,FCoM_envel:ATrPS_freq,`Jitter->F0_abs_dif`:RPDE) testing <- notna[-trainIndex[,1],] %>% select(RatingCategory,FCoM_envel:ATrPS_freq,`Jitter->F0_abs_dif`:RPDE) fitControl <- trainControl( method = "repeatedcv", number = 10, allowParallel=TRUE, repeats=100, savePredictions="final") ordCVFit <- train(RatingCategory ~ ., data = training, method = "ordinalNet", trControl = fitControl, preProcess=c("center", "scale"), metric="Kappa", tuneGrid=expand.grid(alpha=1,criteria="bic", link="logit") )
But then there should be a model evaluation stage, and I then apply the model to predict the data points in the evaluation part (20% above) of the data.
But is this really sufficient? I then have a model and a performance in unknown data, but is actually just relying on this data being representative of all the possible evaluation data sets that I could have constructed in the random sampling.
Is this ok? What else could I do?