# How to select best models if the ROC AUC score changes drastically at each separate run?

Below are two plots for ROC curves with their AUC mentioned in the legend brackets. How do I shortlist the best models if the scores differ at each run?

Should I rather calculate the ROC AUC only from predictions from a cross validation? Here, I have trained the models on a smaller train set and predicted on a holdout set. No cross-validation.

$$N$$-fold cross-validation ameliorates this somewhat by using a different test set each time. However, since $$N-1$$ folds are in common between any two training sets, the estimates of out-of-sample error we obtain during each of the $$N$$ rounds may be highly correlated. This is most extreme and most obvious with leave-one-out cross validation.