3
$\begingroup$

Let's suppose that I have seven classification models (RF, SVM, ANN, DT, kNN, Logistic Reg, LDA), and I would like to evaluate their performance through various thresholds. Receiver operating characteristics (ROC) curve is one of the options out there to do this evaluation. And a certain threshold (let's say the medium threshold) was adopted to tune the models and to select the salient features. Then, a classifier, such as DT (decision trees) could be with the poorest performance at this threshold but it "surprisingly" turns out to have the best ROC curve, which means it is the best over various thresholds (and it also beats SVM!). So... Do I need to tune these classifiers and conduct the feature selection at each threshold when the ROC curves are generated?

$\endgroup$
3
  • $\begingroup$ Choosing the medium threshold seems strange, wouldn’t that result in different false positive rates for the different classifiers? $\endgroup$
    – kbrose
    Commented Apr 27, 2018 at 14:19
  • $\begingroup$ Yes, it would lead to different FPRs (on x-axis). From your opinion, at which threshold the models should be tuned? $\endgroup$
    – mhdella
    Commented Apr 27, 2018 at 20:48
  • $\begingroup$ I would instead choose an acceptable FPR and see which model has the highest TPR $\endgroup$
    – kbrose
    Commented Apr 28, 2018 at 0:32

1 Answer 1

1
$\begingroup$

NO

The point of the ROC curve is to assess model performance across a range of thresholds. In that sense, the ROC curve and the area under it are measures of how well the model is able to separate the two classes. This is separate from applying a threshold and going with the classifications that arise from the two-step model that first makes predictions on a continuum and then uses a threshold to decide on the bin to which each prediction corresponds.

It might be that some thresholds are terrible for your task, and that is okay. If you get to a point where you have to use a threshold, you would just use a different one and be okay with the fact that some thresholds do not work. In fact, if you have a logistic regression that makes probability predictions on the interval $[0,1]$, setting a threshold of $-2$ or $+2$ puts you in a position where every observation is classified the same way no matter what modeling you do, which probably is not acceptable performance. In that situation, there is literally nothing you can do to tune the model to make varied categorical predictions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.