Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Below are two plots for ROC curves with their AUC mentioned in the legend brackets. How do I shortlist the best models if the scores differ at each run?
Should I rather calculate the ROC AUC only from predictions from a cross validation? Here, I have trained the models on a smaller train set and predicted on a holdout set. No cross-validation.
When comparing models, one typically wishes to select the model which has minimal prediction error on unseen data (out-of-sample error); ie, the most "generalisable" model. To do this, it's common to partition data into training and test sets, in order to obtain an estimate of the out-of-sample error. However, the estimate of out-of-sample error we get in this setting is highly dependent on the data chosen to be used in the training and test sets.
$N$-fold cross-validation ameliorates this somewhat by using a different test set each time. However, since $N-1$ folds are in common between any two training sets, the estimates of out-of-sample error we obtain during each of the $N$ rounds may be highly correlated. This is most extreme and most obvious with leave-one-out cross validation.
I would typically never choose a model based on a single train/test split, instead opting for cross-validation. Unless the models are computationally intensive to train, I would in fact choose to perform repeated N-fold cross-validation. Here, we repeat N-fold cross-validation a number of times, resulting in a different N-fold split each time. This results in a more unbiased and lower variance estimate of the out-of-sample error.1
