Let's suppose that I have seven classification models (RF, SVM, ANN, DT, kNN, Logistic Reg, LDA), and I would like to evaluate their performance through various thresholds. Receiver operating characteristics (ROC) curve is one of the options out there to do this evaluation. And a certain threshold (let's say the medium threshold) was adopted to tune the models and to select the salient features. Then, a classifier, such as DT (decision trees) could be with the poorest performance at this threshold but it "surprisingly" turns out to have the best ROC curve, which means it is the best over various thresholds (and it also beats SVM!). So... Do I need to tune these classifiers and conduct the feature selection at each threshold when the ROC curves are generated?

  • $\begingroup$ Choosing the medium threshold seems strange, wouldn’t that result in different false positive rates for the different classifiers? $\endgroup$ – kbrose Apr 27 '18 at 14:19
  • $\begingroup$ Yes, it would lead to different FPRs (on x-axis). From your opinion, at which threshold the models should be tuned? $\endgroup$ – mhdella Apr 27 '18 at 20:48
  • $\begingroup$ I would instead choose an acceptable FPR and see which model has the highest TPR $\endgroup$ – kbrose Apr 28 '18 at 0:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.