5
$\begingroup$

What is the interpretation of the number that is the output of the logistic regression function?

The logistic function

$$f(\vec{x}) = \frac{1}{1+e^{-g(\vec{x})}}$$

(where $g$ is a linear function) is supposed to map a continuous variable (or more generally a whole bunch of totally ordered variables) to between 0 and 1.

I've always assumed it is the probability of inclusion in one set or the other. The range is $[0,1]$ (well, maybe not 0 and 1), which is what a probability is. And frankly anything between 0 and 1, what else could it be other than a probability.

But looking at the curve, I started to doubt. I wondered if it is necessarily to be interpreted as a probability. It looks like a probability but is it really? Just because they share the same range doesn't mean they are the same. If $f(\vec{x}) = .75$, does that really mean that $75\%$ of $f$ is less than $f(\vec{x})$?

This could go in two directions:

  • Suppose it is a probability, or more exactly the probability of a 'true', '1', or 'positive' classification of a point in the domain. How is this justified?

  • Suppose not. Then what is it exactly and why? How far a way is it from a probability (numerically and conceptually)?

Another way to say this is what is so special about $1/(1+e^{-g(x)})$? Why not any monotonically increasing odd (about $y=1/2$) function with the same range, like $$f(x) = \frac{\tan^{-1}(g(x))+\pi/2}{\pi}$$ or $$f(x) = {\rm erf}(g(x)) = \frac{2}{\sqrt{\pi}} \int_{-\infty}^x e^{-t^2} \ {\rm d}t$$ (very close but not equal to the logistic function)

3 sigmoid-like functions

or frankly $$f(x) = \begin{array}{ll} 0,& {\rm if\ } g(x) < 0\\ 1,& {\rm if\ } g(x) >= 0 \end{array}$$ ?

$\endgroup$
  • 3
    $\begingroup$ Logistic regression is supposed to fit the conditional probability $p(Y=1|{\bf X}=\vec{x})$, but of course, it is possible the model is misspecified. $\endgroup$ – gung Dec 3 '15 at 21:01
  • $\begingroup$ @gung but what are the grounds of such assumption? Just the fact that sigmoid value lies between zero and 1? $\endgroup$ – Yurii Dec 3 '15 at 21:29
  • $\begingroup$ @Yurii, what assumption? If the relationship is curvilinear, but you don't include a curvilinear term, the fitted probabilities would differ from the true probabilities. $\endgroup$ – gung Dec 3 '15 at 21:35
  • $\begingroup$ @gung the assumption is that logistic regression is going to produce conditional probability you mentioned $\endgroup$ – Yurii Dec 3 '15 at 21:39
  • $\begingroup$ See also stats.stackexchange.com/questions/163034/… $\endgroup$ – Tim Dec 4 '15 at 15:46
4
$\begingroup$

What is the interpretation of the number that is the output of the logistic regression function?

Logistic regression as understood in recent decades is explicitly used as a model for Bernoulli or binomial data (with extensions into other cases such as multinomial), where the model if for the parameter, $p$, which is indeed a probability.

However, logistic regression has its origins in modelling the growth of a proportion over time[1] (which may be continuous), so in origins it bears a close link to nonlinear models that fit logistic growth curves

And frankly anything between 0 and 1, what else could it be other than a probability.

Well, something between 0 and 1 could be a model a continuous fraction such as the proportion of substance A in a mix of things. Can logistic regression model such a thing? The model for the mean makes sense, but the model for the variance doesn't necessarily make sense; in logistic regression the variance function is of the form $\mu(1-\mu)$. This is directly related to the variance of a Bernoulli.

However (for example) one could consider approximating something like a beta (which has variance function proportional to $\mu(1-\mu)$) by a quasi-binomial model; then we wouldn't necessarily be modelling a probability as such, but we'd still arguably be using logistic regression to do it.

So while it's nearly always conceived as a model for a probability, it doesn't necessarily have to be.

Suppose it is a probability, or more exactly the probability of a 'true', '1', or 'positive' classification of a point in the domain. How is this justified?

I don't understand the question here. If it's explicitly a model for $p$ in a Bernoulli, what additional sort of justification do you seek? Of course the link function may be wrong (while that's no great difficulty - since other links could be used - we would no longer be doing logistic regression).

[1]: Cramer, J.S. (2002),
"The Origins of Logistic Regression,"
Tinbergen Institute, December
http://papers.tinbergen.nl/02119.pdf

$\endgroup$
  • $\begingroup$ I think what I'm trying to get at is why aren't other similarly shaped curves not measures of probability? Like $f(2x) = \frac{\tanh(x)+1}{2}$, or the CDF of the normal curve? $\endgroup$ – Mitch Dec 8 '15 at 16:52
  • 1
    $\begingroup$ Using the CDF of a normal curve is explicitly not logistic regression, as it does not use a logistic function. Models using the normal CDF are called probit. Yet another kind of -obit model is the robit, which uses the Cauchy CDF. $\endgroup$ – Sycorax Dec 8 '15 at 17:40
  • $\begingroup$ @Mitch They are! The probit link function is often used in place of the logit in Bernoulli(/binomial) glms, for example (and it's not the only one; many stats packages offer the complementary-log-log link function, which is another cdf). I'm pretty sure your edited question is now a duplicate though. $\endgroup$ – Glen_b Dec 8 '15 at 22:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.