2
$\begingroup$

IIUC, logistic regression can be thought of as performing linear regression on $ \log(\frac{p}{1-p}) $. When that's the case, it's understandable why $\frac{1}{1+e^{-w^Tx}}$ represents the probability (p) of success. What I'm missing is why it makes sense to view the points ($x_i$,$y_i$) as ($x_i$, $\log(\frac{p_i}{1-p_i}))$ to begin with.

$\endgroup$
  • 2
    $\begingroup$ You probably need to review the generalized linear model's theory a little more. Most of the statements you made in your post are imprecise, if not incorrect. Alan Agresti's classic Categorical Data Analysis would be a good choice. $\endgroup$ – Zhanxiong Aug 13 '17 at 5:26
  • 1
    $\begingroup$ I Am not sure that logistic regression can be seen as lineair regression on the transformed variable that you mention. $\endgroup$ – user83346 Aug 13 '17 at 9:45
  • $\begingroup$ @fcop can you explain why you don't think you can view this in terms of transformed variables? I see no issue with the poster's question. $\endgroup$ – jds Aug 13 '17 at 13:53
  • $\begingroup$ Note: We are not really viewing the points $(x_i, y_i)$ as $(x_i, \log(p_i/(1-p_i))$, more we are dealing with one layer of the hierarchical model implied by logistic-regression as existing on a logit transformed scale (see my answer below). $\endgroup$ – jds Aug 13 '17 at 15:40
  • 2
    $\begingroup$ Logistic regression can't be understood as performing linear regression on $y=\log(p/(1-p))$. Among other things, note that you don't even observe $p$ to be able to transform it into a new dependent variable $y$ in the first place. $\endgroup$ – The Laconic Aug 13 '17 at 16:53
3
$\begingroup$

This is ultimately a subjective question that will come down to "what makes sense to you". That said I will try my best to provide some intuition on the matter.

For a univariate logistic-regression your data is given as $(x_i, y_i)$ where $y_i$ is a binary outcome (i.e., $y_i\in\{0,1\}$). In this case we can write a logistic regression model as $$y_i \sim \text{Bernoulli}(p_i)$$ $$p_i \sim \text{logistic-normal}(\mu, \sigma^2)$$ I will first note that the logistic-normal distribution is not very often discussed. However, I think your question is best framed as the following questions:

  1. Why is the Bernoulli-Logistic-Normal a good Model for my $(x_i, y_i)$ data.
  2. Why does the logit function $\log \left(\frac{p_i}{1-p_i}\right)$ a useful transformation of the $p_i$ variables?

Why the Bernoulli-Logistic-Normal Model?

I think the Bernoulli component is fairly obvious, we have data ($y_i$) that takes on binary values {${0,1}$} and we want to model the probability that it takes on each of these two values $p_i$.

Now why the Logistic-Normal?

Note that our probabilities $p_i$ are not simply $p_i$ but actually the tuple ($p_{i0}, p_{i1}$), that is the probabilities of seeing a $0$ or a $1$ respectively with the linear constraint $p_{i0}+p_{i1}=1$. Think of it like this, does it matter that probabilities sum to 1? Not really, they could sum to 77 if we wanted but what really matters is the "relative" probability of one event (the probability of seeing a 1) to the other event (the probability of seeing a 0).

So what do I mean by relative? Well we could think of relative in terms of the arithmetic difference between probabilities (e.g., $p_{i1}-p_{i0}$) but this really doesn't end up working out in practice and it doesn't really end up making much sense. For one thing, if we change probabilities from summing to 1 to summing to some arbitrary total $k$, our value for $p_{i1}-p_{i0}$ changes as well, this doesn't reflect the idea that the sum is arbitrary but what really matters is the relative probabilities of the events. A better option is $p_{i1}/p_{i0}$ (i.e., the ratio of the probabilities). Notice that the ratio is invariant to what we choose as the total sum of probabilities and the ratio transforms the constrained $[0,1]$ space of probabilities to the $[0,+\infty)$ which moves us closer to working with values in real space rather than a constrained subset of real space (its often difficult to work with constrained subsets of real space). However, now we are working with ratios (on a multiplicative scale) and this is not so easy. One thing we can do to make this additive is to take the log such that $$\log(p_{i1}/p_{i0}) = \log(p_{i1}) - \log(p_{i0}).$$

Notice that this log-ratio is identical to the logit transform ($\log(p_{i1}/p_{i0}) = \log(p_i / (1-p_i))$). Also notice that by taking the log-ratio we have transformed our constrained $[0,1]$ scale to $(-\infty, +\infty)$ (however the cost is that we have had to "chop-off" the boundaries - but this is a side note).

So I have somewhat "hand-wavingly" motivated the multiplicative (or ratio) scale of probabilities and the logit-transform. But I have been slow to get at the Logistic-Normal which I said I was going to tackle. Well it turns out that if our errors are additive on a log-ratio scale then the central limiting distribution is a logistic-normal. All this is to say that the Logistic-Normal is a good choice (and somewhat analogous to the normal distribution) for probabilities.

Why the Logit Transform?

I have already given some motivation for the logit-transformation as linearizing multiplication (or ratios) and transforming a constrained $[0,1]$ subset of real space to the unconstrained real line $(-\infty, +\infty)$. However, these statements may seem too abstract. If instead we accept the utility of the logistic-normal distribution we can then see the logit-transformation as just an easy way to work with the logistic-normal.

The logistic-normal as it is traditionally parameterized (with respect to the Lebesgue measure - don't worry if this comment about measure does not make sense to you) is really difficult to work with. Notice that on wikipedia the logistic-normal (aka. logit-normal) has no analytical solution for the mean or variance.

Well it turns out that we can transform logit-normal random variables to normal random variables through a logit-transformation (or an inverse-logit transformation depending on how you write it) making it much easier to work with.

$$y_i \sim \text{Bernoulli}(p_i)$$ $$p_i = logit^{-1}(\eta_i)$$ $$\eta_i \sim \text{normal}(\mu, \sigma^2).$$

One final comment

In case you find this discussion too abstract or unappealing. Consider that the logit scale (e.g., $p_i/(1-p_1)$ is essentially just the "log-odds" scale.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.