Where does the binary logistic regression model equation come from?

Question

In Frank Harrell's Regression Modeling Strategies, he states:

The ordinary linear regression model is:

$$C(Y|X)=E(Y|X)=X\beta$$

and given $X$, $Y$ has a normal distribution with mean $X\beta$ and constant variance $\sigma^2$. The binary logistic regression model is:

$$C(Y|X)=\textrm{Prob}(Y=1|x)=(1+\exp(-X\beta))^{-1}$$

How is this formula $(1+\exp(-X\beta))^{-1}$ derived? I have tried looking at his cited sources but it is still not clear to me.

How do we go from $C(Y|X)=E(Y|X)=X\beta$ to $\textrm{Prob}(Y =1|X)=(1+\exp(-X\beta))^{-1}$?

Answer 1

I wouldn’t say it was “derived”, but rather designed. In generalized linear models

$$C(Y|X)=E(Y|X)=X\beta$$

$C$ is a link function. For linear regression its inverse, $C^{-1}$, is an identity function; for logistic regression it’s the logit function. $Y$ is assumed to follow a Bernoulli distinction parametrized by probability of success $p$, that is also its mean. Since probability is bounded between zero and one, we need to transform it to such a range: logit function is one such transformation, probit is another, and there are some other possible choices.

I don’t have the book to hand, but would say it should be

$$ E[Y|X] = C^{-1}(X\beta) $$

and

$$ C(Y|X) = X\beta $$

Answer 2

1 Convenient transformation

The logistic function is often used as a mapping from $(-\infty,\infty)$ to $(0,1)$ (as others mention).

However the logistic function as link function also relates to being the canonical link function, or sometimes it relates to a particular mechanism/model. See the two points below.

2 Canonical link function

In short: the logit of the mean, $\log \left( \frac{p}{1-p} \right) $, is the natural parameter of the Bernoulli distribution. The logistic function is the inverse.

---

You derive it as follows:

The logit/logistic function relates to the Bernoulli/binary when you express the pdf as an exponential family in canonical form, ie when you use as parameter $\theta$ the natural parameter such that $\eta(\theta) = \theta$:

$$f(y\vert \theta) = h(y)e^{\eta(\theta) t(y) - A(\theta)} = h(y)e^{\theta t(y)- A(\theta)}$$

In the case of the binomial distribution the natural parameter is not the probability $p$ (or $\mu$ which equals $p$), which we typically use, but $\eta = \log \left( \frac{p}{1-p} \right)$

$$f(y\vert p) = e^{\log \left(\frac{p}{1-p}\right)y + \log(1-p)}$$

Then the linear function $X\beta$ is used to model this natural parameter:

$$\eta = \log \left( \frac{p}{1-p} \right) = X\beta$$

If we rewrite it such that $p$ is a function of $X\beta$, then you get

$$p = (1-e^{-X\beta})^{-1}$$

So the logistic function $p=(1-e^{-X\beta})^{-1}$ is the inverse of the logit function $X\beta =\log \left( \frac{p}{1-p} \right)$. The latter pops up in the equation above when we write the model with the natural parameter.

3 Growth model or other differential equation relationship

The above, canonical link function, is an afterthought, and the history of the logistic function is older than when it was recognized as canonical link function. The use of a canonical link function can have advantages but there is no reason that the natural parameter needs to be some linear function.

An alternative reason for the use of the link function can be when it actually makes sense as a deterministic model. For instance in growth models the logistic function can arrise.

When the growth equals

$$f'= f(1-f)$$

Then the solution is the logistic function. You can see the above as exponential growth when $1-f\approx 1$ that becomes limited when $f$ approaches $1$.

Answer 3

To me, this paper from John Mount was instructive. He derives the logistic regression formula using two approaches, one them using the maximum entropy principle.

Answer 4

You obtain the sigmoid function by making the assumption that a linear combination of your inputs gives you the log-odds of the two classes. That is the log of the ratio of the probabilities of class $1$ to class $0$, $$ X \beta = \log\left(\frac{p_1}{p_0}\right) = \log\left(\frac{p_1}{1-p_1}\right). $$

This is an assumption taken from scratch, similar to the linear regression assumption that the expected output is directly a linear combination of the inputs.The reason the log-odds is a common choice for the linear quantity is that its range is $(-\infty,\infty)$. You can see that the limit of the above function as $p_1 \rightarrow 0$ is $-\infty$, and as $p_1 \rightarrow 1$ it approaches $+\infty$. A linear combination of arbitrary inputs is an unbounded continuous number, so the target that it's modelling must also represent an unbounded continuous number.

It's easy to show that the inverse of the above expression is $$ p_1 = \frac{1}{1 + \exp(-X \beta)}. $$

Answer 5

Contrary to some of the answers in this thread, I would like to give a derivation of the formula which I like.

Suppose we have a random variable that can take either of two classes $C_1$ or $C_2$. We are interested in finding the probability of $C_k$ conditioned on some observation $x$, i.e., we want to estimate $p(C_k\vert x)$. To model this, consider the following:

Using Baye's Theorem we have that

$$ \begin{aligned} p(C_1\vert x) &= \frac{p(C_1)p(x\vert C_1)}{p(C_1)p(x\vert C_1) + p(C_2)p(x\vert C_2)}\\ &= \frac{p(C_1)p(x\vert C_1)}{p(C_1)p(x\vert C_1) + p(C_2)p(x\vert C_2)} \frac{(p(C_1)p(x\vert C_1))^{-1}}{(p(C_1)p(x\vert C_1))^{-1}}\\ &= \frac{1}{1 + \frac{p(C_2)p(x\vert C_2)}{p(C_1)p(x\vert C_1)}}\\ &= \frac{1}{1 + \exp\left(\log\left(\frac{p(C_2)p(x\vert C_2)}{p(C_1)p(x\vert C_1)}\right)\right)}\\ &= \frac{1}{1 + \exp\left(-\log\left(\frac{p(C_1)p(x\vert C_1)}{p(C_2)p(x\vert C_2)}\right)\right)}\\ \end{aligned} $$

Denoting $z(x)=\log\left(\frac{p(C_1)p(x\vert C_1)}{p(C_2)p(x\vert C_2)}\right)$, we arrive at the formula:

$$ p(C_1\vert x) = \frac{1}{1 + \exp(-z(x))} $$

In a logistic regression, we are assuming the existence of a vector $\boldsymbol\beta\in\mathbb{R}^M$ of weights such that $\boldsymbol\beta^T\phi(x) = \log\left(\frac{p(C_1)p(x\vert C_1)}{p(C_2)p(x\vert C_2)}\right)$, for some function $\phi:\mathbb{R}\to\mathbb{R}^M$ known as the basis function. That is, assuming that the latter is true, then the conditional probability $p(C_1\vert x)$ is given by

$$ p(C_1\vert x) = \frac{1}{1 + \exp(-\boldsymbol\beta^T\phi(x))} $$

---

On a personal note, I believe that it is a bold claim to state that $\boldsymbol\beta^T\phi(x) = \log\left(\frac{p(C_1)p(x\vert C_1)}{p(C_2)p(x\vert C_2)}\right)$. I do not see how it is trivial to argue that this has to be the case. In fact, by modeling the factor $z(x)$ disregarding the underlying distributions is known as a discriminative model. If we want to model explicitly the terms for $z(x)$ we would have a generative model.