6
$\begingroup$

My question is motivated in part by the possibilities afforded by scikit-learn. In the documentation, there are two examples of how to compute a Receiver Operating Characteristic (ROC) Curve.

One uses predict_proba to

Compute probabilities of possible outcomes for samples [...].

, while the other uses decision_function, which yields the

Distance of the samples X to the separating hyperplane.

When should each one be used?

$\endgroup$
  • 2
    $\begingroup$ stackoverflow.com/questions/36543137/… $\endgroup$ – Nikolas Rieble Jan 5 '17 at 14:07
  • 2
    $\begingroup$ Be aware that questions should be self-contained. We don't want someone to have to navigate to a link & read the page to understand what you are asking. Moreover, most people on CV are unlikely to automatically know what ideas are being referenced by predict_proba & decision_function, so people won't necessarily know that this is about 2 different strategies, or what the strategies are. $\endgroup$ – gung Jan 5 '17 at 18:08
  • $\begingroup$ @gung: (in reply to the last comment) Thank you, very good point. It's hopefully more self-contained now. It was about the middle ground: which method to use, but most importantly why. Hopefully it's ok'ish now. There is no need for apologies, as your comments so far helped to make it more appropriate $\endgroup$ – serv-inc Jan 5 '17 at 18:26
  • 1
    $\begingroup$ I am retracting my close vote. I now believe this question is on topic here. $\endgroup$ – gung Jan 5 '17 at 18:52
4
$\begingroup$

I am answering this from a pragmatic perspective, simply by looking at code and deducing from examples. A more theoretic answer could be a great supplement.

Generally both can be used. The difference is well explained here.

Yet most relevant, not all algorithms offer both predict_proba and decision_function. To my knowledge, every classifiers allows predict_proba in sklearn. For some - specifically SVC (Support Vector Classification) - both give exactly the same result. To check, I used this example and change the code once using predict_proba and once decision_function.

Specifically I changed:

probas_ = classifier.fit(X[train], y[train]).predict_proba(X[test])
# Compute ROC curve and area the curve
fpr, tpr, thresholds = roc_curve(y[test], probas_[:, 1])

to:

probas_ = classifier.fit(X[train], y[train]).decision_function(X[test])
# Compute ROC curve and area the curve
fpr, tpr, thresholds = roc_curve(y[test], probas_)   

And both yield the exact same result as you can see in the images:

enter image description here

enter image description here

Yet this only counts for SVC where the distance to the decision plane is used to compute the probability - therefore no difference in the ROC.

In another example a specific line of code is relevant for this question:

if hasattr(clf, "decision_function"):
    Z = clf.decision_function(np.c_[xx.ravel(), yy.ravel()])
else:
    Z = clf.predict_proba(np.c_[xx.ravel(), yy.ravel()])[:, 1]

Therefore, deducing from the sklearn example, I would recommend you use the decision_function wherever possible and if not, use the probability provided by predict_proba.

Examples for algorithms which do not provide a decision_function in sklearn:

  • KNeighborsClassifier()
  • RandomForestClassifier()
  • GaussianNB()
$\endgroup$
  • $\begingroup$ Why do you make that recommendation? Based on that line of code from an example? It's worth noting that if the probability is a monotonic function of the decision score (as it generally should be), then the results will be the same. $\endgroup$ – Matthew Drury Jan 31 '18 at 13:53
  • $\begingroup$ Good point. I tried to answer just deducing from code as emphasized in the answer. Basically the answer is specific to the package since i derive the recommendation from the source code, where the developers use the decision function with priority. $\endgroup$ – Nikolas Rieble Jan 31 '18 at 16:17
  • 1
    $\begingroup$ Over time, I've grown more and more suspicious of the sklearn developers decision making on things like this. This is a pretty clear case of the wrong advice, as a decision rule that is not monotonic with respect to probabilities is inconsistent. If the probability is a monotonic function of the decision function this is fine, but it still makes more conceptual sense to threshold the probabilities. The only defense I can think of is if computation of the probability from the decision function is expensive. $\endgroup$ – Matthew Drury Jan 31 '18 at 16:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.