I have been going through the sklearn documentation but I am not able to understand the purpose of these functions in the context of logistic regression.
For decision_function it says that its the distance between the hyperplane and the test instance. how is this particular information useful? and how does this relate to predict and predict-proba methods?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
10
Recall that the functional form of logistic regression is
$$ f(x) = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k)}} $$
This is what is returned by predict_proba.
The term inside the exponential
$$ d(x) = \beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k $$
is what is returned by decision_function. The "hyperplane" referred to in the documentation is
$$ \beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k = 0 $$
This terminology is a holdover from support vector machines, which literally estimate a separating hyperplane. For logistic regression this hyperplane is a bit of an artificial construct, it is the plane of equal probability, where the model has determined both target classes are equally likely.
The predict function returns a class decision using the rule
$$ f(x) > 0.5 $$
At the risk of soapboxing, the predict function has very few legitimate uses, and I view using it as a sign of error when reviewing others work. I would go far enough to call it a design error in sklearn itself (the predict_proba function should have been called predict, and predict should have been called predict_class, if anything at all).
answered Feb 21, 2018 at 18:01
-
$\begingroup$ Thanks for the answer @Matthew, but can you clarify this point a bit more "For logistic regression, this hyperplane is a bit of an artificial construct, it is the plane of equal probability, where the model has determined both target classes are equally likely." ? $\endgroup$– SameedFeb 24, 2018 at 15:02
-
$\begingroup$ This explanation is interesting and helpful. I wish sklearn explained it better. What I don't understand is what is the use of knowing the value of x in the logistic function 1/(1+e^-x)? All I can think of is to possibly use a different sigmoid function like x/(1+|x|). Is there more? thanks! $\endgroup$– ldmtwoApr 20, 2018 at 17:28
-
$\begingroup$ Basically the decision function should have been the sigmoid in the logistic regression. Correct? $\endgroup$– 3nomisOct 3, 2019 at 19:31
-
3$\begingroup$ I think the reason for @Matthew being on a soapbox is that using 0.5 as the threshold for prediction is naive. The first thing one should do is learn to use cross-validation, ROC curves and AUC to choose an appropriate threshold c, and using as the decision function f(x) > c. $\endgroup$– hwrdMar 4, 2020 at 22:21
-
2$\begingroup$ @Mathew Drury. I computed
1/(1+np.exp(-lr.decision_function(X)), but the result does not match exactly tolr.predict_proba. They are close, but not the same. Do you know why? $\endgroup$– SarahAug 12, 2020 at 16:50