24
$\begingroup$

I have been going through the sklearn documentation but I am not able to understand the purpose of these functions in the context of logistic regression. For decision_function it says that its the distance between the hyperplane and the test instance. how is this particular information useful? and how does this relate to predict and predict-proba methods?

$\endgroup$

1 Answer 1

53
$\begingroup$

Recall that the functional form of logistic regression is

$$ f(x) = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k)}} $$

This is what is returned by predict_proba.

The term inside the exponential

$$ d(x) = \beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k $$

is what is returned by decision_function. The "hyperplane" referred to in the documentation is

$$ \beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k = 0 $$

This terminology is a holdover from support vector machines, which literally estimate a separating hyperplane. For logistic regression this hyperplane is a bit of an artificial construct, it is the plane of equal probability, where the model has determined both target classes are equally likely.

The predict function returns a class decision using the rule

$$ f(x) > 0.5 $$

At the risk of soapboxing, the predict function has very few legitimate uses, and I view using it as a sign of error when reviewing others work. I would go far enough to call it a design error in sklearn itself (the predict_proba function should have been called predict, and predict should have been called predict_class, if anything at all).

$\endgroup$
10
  • $\begingroup$ Thanks for the answer @Matthew, but can you clarify this point a bit more "For logistic regression, this hyperplane is a bit of an artificial construct, it is the plane of equal probability, where the model has determined both target classes are equally likely." ? $\endgroup$
    – Sameed
    Feb 24, 2018 at 15:02
  • $\begingroup$ This explanation is interesting and helpful. I wish sklearn explained it better. What I don't understand is what is the use of knowing the value of x in the logistic function 1/(1+e^-x)? All I can think of is to possibly use a different sigmoid function like x/(1+|x|). Is there more? thanks! $\endgroup$
    – ldmtwo
    Apr 20, 2018 at 17:28
  • $\begingroup$ Basically the decision function should have been the sigmoid in the logistic regression. Correct? $\endgroup$
    – 3nomis
    Oct 3, 2019 at 19:31
  • 4
    $\begingroup$ I think the reason for @Matthew being on a soapbox is that using 0.5 as the threshold for prediction is naive. The first thing one should do is learn to use cross-validation, ROC curves and AUC to choose an appropriate threshold c, and using as the decision function f(x) > c. $\endgroup$
    – hwrd
    Mar 4, 2020 at 22:21
  • 2
    $\begingroup$ @Mathew Drury. I computed 1/(1+np.exp(-lr.decision_function(X)), but the result does not match exactly to lr.predict_proba. They are close, but not the same. Do you know why? $\endgroup$
    – Sarah
    Aug 12, 2020 at 16:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.