How to select a model based on ROC AUC, sensitivity and specificity?

Question

I'm running several machine learning algorithms on a dataset with 80% negatives and 20% positive cases (classification). Below I attach the results of comparing performance on 500 bootstrap resamples for five methods

where NB=Naive Bayes, RF=Random Forest, KNN= K-Neares Neighbors and SVM= Support vector machine (linear kernel). My question is, based on this results, which model would we say is best to analyse our dataset? I read the general recommendation is to go for AUC-ROC, but in my case we also have:

• A very precise estimate of the prevalence (this is a disease for which positives are known to be 20%)

• The cost of a false negative is much higher than that of a false positive

For these two reasons, could we say that Sensitivity or Positive Predictive Value are better than AUC-ROC? Is SVM better here than NB?

I also understand that higher ROC means that for a different threshold there might be a higher sensitivity, but I don't know how to select one without being arbitrary or incurring in selection bias.

Can I have your opinions? Thanks!

Answer 1

First of all, the performances displayed in the boxplots have a huge variance. So maybe you want to explain in more detail how you created the bootstrap samples and those results. I assume you used the bootstrap samples to train the model and the boxplots display the results on the test set.

Optimising the Recall/Sensitivity is not ideal because you run into the same problem as if you would be using accuracy. You can simply get maximum recall by labelling everything as positive cases. The only model that seems to perform somehow okay is the KNN because it has constantly a non-zero ROC-AUC. You can simply choose on the ROC-curve a threshold that gives you a high sensitivity, which results in a low False Negative Rate.

The choice of threshold is not a form of selection bias. It is actually inevitable to decide for one. The models give you a probability for belonging to the positive class in this sort of classification problems. This is the reason why we use the ROC because it is invariant to a possible choice of a specific threshold. The ROC examines the performace for all possible thresholds and we can choose the point on the curve that is most optimal for our problem at hand.