0
$\begingroup$

Below are two plots for ROC curves with their AUC mentioned in the legend brackets. How do I shortlist the best models if the scores differ at each run?

Should I rather calculate the ROC AUC only from predictions from a cross validation? Here, I have trained the models on a smaller train set and predicted on a holdout set. No cross-validation.

enter image description here

$\endgroup$

1 Answer 1

1
$\begingroup$

When comparing models, one typically wishes to select the model which has minimal prediction error on unseen data (out-of-sample error); ie, the most "generalisable" model. To do this, it's common to partition data into training and test sets, in order to obtain an estimate of the out-of-sample error. However, the estimate of out-of-sample error we get in this setting is highly dependent on the data chosen to be used in the training and test sets.

$N$-fold cross-validation ameliorates this somewhat by using a different test set each time. However, since $N-1$ folds are in common between any two training sets, the estimates of out-of-sample error we obtain during each of the $N$ rounds may be highly correlated. This is most extreme and most obvious with leave-one-out cross validation.

I would typically never choose a model based on a single train/test split, instead opting for cross-validation. Unless the models are computationally intensive to train, I would in fact choose to perform repeated N-fold cross-validation. Here, we repeat N-fold cross-validation a number of times, resulting in a different N-fold split each time. This results in a more unbiased and lower variance estimate of the out-of-sample error.1

$\endgroup$
1
  • 1
    $\begingroup$ Thank you. That does answer my question. (I'll delete this comment later) $\endgroup$ Sep 16, 2020 at 4:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.