4
$\begingroup$

Why do we interpret the results of logistic regression as probabilities? Passing the output of any regression procedure through a sigmoid function results in a probabilistic interpretation with respect to classification. Why is that so? Given that the output is between 0 and 1, is it enough to interpret the results as probabilities?

$\endgroup$
  • 4
    $\begingroup$ Logistic regression is (explicitly) a model for $P(Y=1|\mathbf{X}=\mathbf{x})$. $\endgroup$ – Glen_b Sep 28 '14 at 9:12
  • $\begingroup$ Also, if you train the model using some objective function derived from probability theory ( Maximum Likelihood, for example), then the output of the model may be interpreted as a calibrated probability. $\endgroup$ – mcvz Oct 2 '14 at 9:20
1
$\begingroup$

Why do we interpret the results of logistic regression as probabilities?

Because the logistic regression model can be viewed as arising from a linear regression latent variable model, where the error term of this linear regression is assumed to follow the standard logistic distribution. See for example this post.

Given that the output is between 0 and 1, is it enough to interpret the results as probabilities?

No. The "output" must come from a function that satisfies the properties of a distribution function in order for us to interpret it as probabilities. These properties are:

1) The function $F$ under consideration must be non-decreasing and right-continuous ("cadlag")

2) $\lim_{x\rightarrow -\infty}F(x) =0$

3) $\lim_{x\rightarrow \infty}F(x) =1$

The "sigmoid function" satisfies these properties.

$\endgroup$
  • $\begingroup$ The latent variable representation is of logistic regression is common in some fields (economics jumps to mind) but not in all fields. I would say that this justification is one of two possible justifications, the other being "the expected proportion of $1$s interpreted as probability" justification. $\endgroup$ – Maarten Buis Oct 1 '14 at 18:44
  • $\begingroup$ @MaartenBuis Certainly, although I don't see them as "alternatives", since the latent-variable repr. being one level deeper, has necessarily to be interpreted eventually as "expected proportion of $1$s...", at the level of the logistic regr. itself. The latent-var. approach is helpful in that it provides a statistical framework for the choice of the upper-level model. Say, if an indicator function is indeed a threshold-signal from a situation where disturbances exist that are believed to be uniformly distributed, then we should use the Linear Prob. model instead of the logit or probit ones. $\endgroup$ – Alecos Papadopoulos Oct 1 '14 at 19:05
  • $\begingroup$ The latent varialbe interpretation can indeed be very cute. However, in my applied work I have never been in a situation where it has been worth while. In those cases it is extra baggage that distracts from the real message. $\endgroup$ – Maarten Buis Oct 2 '14 at 19:05
1
$\begingroup$

A probability is bounded between 0 and 1 (inclusive), and a sigmoid curve is a convenient curve that can be forced to respect those bounds. It is not the only one, but the sigmoid curve has proven to be the most popular one. So, not all probability models have a sigmoid cure, though many do.

Moreover, not all models with a sigmoid curve model probabilities. For example, models that model a dependent variable that is a fraction also often use a sigmoid curve.

What makes logit, probit, and similar models model a probability is the fact that they model the conditional mean of an indicator variable, which is the conditional proportion of $1$s, which in turn is interpreted as the probability of having a $1$ on that indicator variable.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.