I'm running several machine learning algorithms on a dataset with 80% negatives and 20% positive cases (classification). Below I attach the results of comparing performance on 500 bootstrap resamples for five methods
where NB=Naive Bayes, RF=Random Forest, KNN= K-Neares Neighbors and SVM= Support vector machine (linear kernel). My question is, based on this results, which model would we say is best to analyse our dataset? I read the general recommendation is to go for AUC-ROC, but in my case we also have:
A very precise estimate of the prevalence (this is a disease for which positives are known to be 20%)
The cost of a false negative is much higher than that of a false positive
For these two reasons, could we say that Sensitivity or Positive Predictive Value are better than AUC-ROC? Is SVM better here than NB?
I also understand that higher ROC means that for a different threshold there might be a higher sensitivity, but I don't know how to select one without being arbitrary or incurring in selection bias.
Can I have your opinions? Thanks!