9
$\begingroup$

I'm studying the ROC Curve, and I was wondering if there is any classification algorithm that doesn't return the output class as a result of a certain threshold from the probabilities of the algo?

Because if there is one, how could you have a ROC Curve if you can't use thresholds to draw it, as it gives the output class as a certainty?

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ SVM doesn't, for one. $\endgroup$
    – user2974951
    Commented Feb 21, 2020 at 14:01
  • 2
    $\begingroup$ @user2974951 SVMs have the signed distance to the margin, which are usually used to draw ROCs. $\endgroup$
    – Firebug
    Commented Feb 21, 2020 at 14:10
  • $\begingroup$ Naïve Bayes with dependant variables gives also uncalibrated probabilities. $\endgroup$
    – Samos
    Commented Feb 25, 2020 at 22:03

2 Answers 2

Reset to default
16
$\begingroup$

TLDR: probabilities are not required to build a ROC curve, only a numerical scale supporting the decision.

I'm studying the ROC Curve, and I was wondering if there is any classification algorithm that doesn't return the output class as a result of a certain threshold from the probabilities of the algo?

I previously let this question slip because I focused on what's the actual problem. Many algorithms do not output probabilities at all (it's one of their main selling points actually). SVMs and K-NNs, for example.

Below I'll explain why this is not a problem to build a ROC curve.

Because if there is one, how could you have a ROC Curve if you can't use thresholds to draw it, as it gives the output class as a certainty?

If your algorithm does not give you any other numerical scale of support for the decision, then your ROC curve has only one point.

It's not a wrong ROC, per se, but its usefulness is dubious.

So I'd say that if you don't have this scale (continuous or not), then you can't draw a ROC curve.

Luckly, most algorithms do have this scale. In SVMs it's the distance to the margin, in logistic regression it's the output probability, in decision trees it's the leaf probability, in K-NNs it's the neighborhood voting proportions, etc.

Improve this answer
$\endgroup$
10
$\begingroup$

Support Vector Machines and $k$-Nearest Neighbors come to mind.

(See here for a motivation for short answers. Longer answers are always welcome.)

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ Because if there is one, how could you have a ROC Curve if you can't use thresholds to draw it, as it gives the output class as a certainty? $\endgroup$
    – user2974951
    Commented Feb 21, 2020 at 14:04
  • 5
    $\begingroup$ SVM gives a distance to a hyper plane. ROC curves only care about tanks. You can order distances, so you can make a ROC curve for an SVM. $\endgroup$
    – Sycorax ♦
    Commented Feb 21, 2020 at 14:58
  • 1
    $\begingroup$ k NN can return "probability" or scores at a granularity of 1/k, though. $\endgroup$
    – cbeleites
    Commented Feb 21, 2020 at 22:34
  • 1
    $\begingroup$ * And we only care about the ranking, not probabilities, because a ROC curve is solely a rank-based method, so any monotonic transformation of the SVM's output of distances (Platt scaling, isotonic regression) that does not change the sign of the distances, will not change the ROC curve. (A monotonic transformation that changes the sign, such as negating all signed distances, will "flip" the ROC curve.) $\endgroup$
    – Sycorax ♦
    Commented Feb 22, 2020 at 14:48
  • $\begingroup$ sklearn KNeighborsClassifier has predict_proba method, it returns class proportions that are of the specified number of closest neighbors. Isn't that predicting probabilities of an observation being of a class? $\endgroup$
    – Akavall
    Commented Feb 22, 2020 at 19:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.