Is it better to compute a ROC curve using predicted probabilities or distances from the separating hyperplane?

My question is motivated in part by the possibilities afforded by scikit-learn. In the documentation, there are two examples of how to compute a Receiver Operating Characteristic (ROC) Curve.

One uses predict_proba to

Compute probabilities of possible outcomes for samples [...].

, while the other uses decision_function, which yields the

Distance of the samples X to the separating hyperplane.

When should each one be used?

• stackoverflow.com/questions/36543137/… – Nikolas Rieble Jan 5 '17 at 14:07
• Be aware that questions should be self-contained. We don't want someone to have to navigate to a link & read the page to understand what you are asking. Moreover, most people on CV are unlikely to automatically know what ideas are being referenced by predict_proba & decision_function, so people won't necessarily know that this is about 2 different strategies, or what the strategies are. – gung Jan 5 '17 at 18:08
• @gung: (in reply to the last comment) Thank you, very good point. It's hopefully more self-contained now. It was about the middle ground: which method to use, but most importantly why. Hopefully it's ok'ish now. There is no need for apologies, as your comments so far helped to make it more appropriate – serv-inc Jan 5 '17 at 18:26
• I am retracting my close vote. I now believe this question is on topic here. – gung Jan 5 '17 at 18:52

I am answering this from a pragmatic perspective, simply by looking at code and deducing from examples. A more theoretic answer could be a great supplement.

Generally both can be used. The difference is well explained here.

Yet most relevant, not all algorithms offer both predict_proba and decision_function. To my knowledge, every classifiers allows predict_proba in sklearn. For some - specifically SVC (Support Vector Classification) - both give exactly the same result. To check, I used this example and change the code once using predict_proba and once decision_function.

Specifically I changed:

probas_ = classifier.fit(X[train], y[train]).predict_proba(X[test])
# Compute ROC curve and area the curve
fpr, tpr, thresholds = roc_curve(y[test], probas_[:, 1])

to:

probas_ = classifier.fit(X[train], y[train]).decision_function(X[test])
# Compute ROC curve and area the curve
fpr, tpr, thresholds = roc_curve(y[test], probas_)

And both yield the exact same result as you can see in the images:  Yet this only counts for SVC where the distance to the decision plane is used to compute the probability - therefore no difference in the ROC.

In another example a specific line of code is relevant for this question:

if hasattr(clf, "decision_function"):
Z = clf.decision_function(np.c_[xx.ravel(), yy.ravel()])
else:
Z = clf.predict_proba(np.c_[xx.ravel(), yy.ravel()])[:, 1]

Therefore, deducing from the sklearn example, I would recommend you use the decision_function wherever possible and if not, use the probability provided by predict_proba.

Examples for algorithms which do not provide a decision_function in sklearn:

• KNeighborsClassifier()
• RandomForestClassifier()
• GaussianNB()
• Why do you make that recommendation? Based on that line of code from an example? It's worth noting that if the probability is a monotonic function of the decision score (as it generally should be), then the results will be the same. – Matthew Drury Jan 31 '18 at 13:53
• Good point. I tried to answer just deducing from code as emphasized in the answer. Basically the answer is specific to the package since i derive the recommendation from the source code, where the developers use the decision function with priority. – Nikolas Rieble Jan 31 '18 at 16:17
• Over time, I've grown more and more suspicious of the sklearn developers decision making on things like this. This is a pretty clear case of the wrong advice, as a decision rule that is not monotonic with respect to probabilities is inconsistent. If the probability is a monotonic function of the decision function this is fine, but it still makes more conceptual sense to threshold the probabilities. The only defense I can think of is if computation of the probability from the decision function is expensive. – Matthew Drury Jan 31 '18 at 16:26