Where does the logistic function come from?

Question

I first learned the logistic function in machine learning, where it is just a function that map a real number to 0 to 1. We can use calculus to get the derivative and use it for some optimization eventually to have some binary classifier.

Later, I learned it in statistic literature where there are odds, log odds, and bunch of probabilistic interpretations.

Today I am reviewing some differential equation literature, and found the logistic function is a solution for the following system:

$y'=y(1-y)$, and $y(0)=0.5$.

---

So, my question is more like a "chicken and egg" problem: where does the logistic function come from? Do we first have the function or the ODE?

Answer 1

I read from Strogatz's book that it was originated from modeling the human populations by Verhulst in 1838. Assume the population size is $N(t)$, then the per capita growth rate is $\dot N(t)/N(t)$. By assuming the per capita growth rate descreases linearly with the population size, we can have the logistic equation of following form: $$\dot N(t)=rN(1-\frac{N}{K}),$$ where $K$ is carrying capacity of the environment. From the equation, we can see that when $N$ is very small, the population grows approximately exponentially. As $N$ grows until half of the capacity, the derivative $\dot N$ is still increasing but slows down. Once $N$ passes the half line, the derivative decreases so we can see a bending curve (Interestingly, we can see this trend very roughly from the bended curves of cumulative Coronavirus cases) and the population asymptotically approaches the capacity. Up to this point, we observe the properties of a logistic function. Intuitively, when $N$ is larger the capacity, the population decreases. By further mathematical simplification of the equation, we have an equation like this: $$\frac{df(x)}{dx}=f(x)(1-f(x)),$$ which has analytical solotion as: $$f(x)=\frac{e^x}{e^x + C}.$$ With $C=1$ we have the logistic function.

Answer 2

I don't know about its history, but logistic function has a property which makes it attractive for machine learning and logistic regression:

If you have two normally distributed classes with equal variances, then the posterior probability of an observation to belong to one of these classes is given by the logistic function.

First, for any two classes $A$ and $B$ it follows from the Bayesian formula:

$$ P(B | x) = \frac{P(x | B) P(B)}{P(x)} = \frac{P(x | B) P(B)}{P(x | A) P(A) + P(x | B) P(B)} = \frac{1}{1 + \frac{P(x | A)P(A)} {P(x | B)P(B)}}. $$

If $x$ is continuous, so that the classes can be described by their PDFs, $f_A(x)$ and $f_B(x)$, the fraction $P(x | A) / P(x | B)$ can be expressed as:

$$ \frac{P(x | A)} {P(x | B)} = \lim_{\Delta x \rightarrow 0} \frac{f_A(x) \Delta x}{f_B(x) \Delta x} = \frac{f_A(x)}{f_B(x)}. $$

If the two classes are normally distributed, with equal variances:

$$ f_A(x) = \frac{1}{\sqrt{2 \pi} \sigma} \exp \left( -\frac{(x - \mu_A)^2} {2 \sigma^2} \right), ~ ~ ~ ~ ~ ~ ~ ~ ~ f_B(x) = \frac{1}{\sqrt{2 \pi} \sigma} \exp \left( -\frac{(x - \mu_B)^2} {2 \sigma^2} \right) $$

then the fraction $f_A(x) / f_B(x)$ can be written as:

$$ \frac{f_A(x)}{f_B(x)} = \exp \left( - \frac{(x - \mu_A)^2} {2 \sigma^2} + \frac{(x - \mu_B)^2} {2 \sigma^2} \right) = \exp \left( \frac{\mu_B^2 - \mu_A^2} {2 \sigma^2} + \frac{\mu_A - \mu_B} {\sigma^2} x \right), $$

and the whole term

$$ \frac{f_A(x)P(A)}{f_B(x)P(B)} = \exp \left( \ln \frac{P(A)}{P(B)} + \frac{\mu_B^2 - \mu_A^2} {2 \sigma^2} + \frac{\mu_A - \mu_B} {\sigma^2} x \right). $$

Denoting

$$ \beta_0 = \frac{\mu_A^2 - \mu_B^2} {2 \sigma^2} - \ln \frac{P(A)}{P(B)} ~ ~ ~ ~ ~ ~ ~ ~ \text{and} ~ ~ ~ ~ ~ ~ ~ ~  \beta_1 = \frac{\mu_B - \mu_A} {\sigma^2} $$

leads to the form commonly used in logistic regression:

$$ P(B | x) = \frac{1}{1 + \exp \left(-\beta_0 - \beta_1 x \right) }. $$

So, if you have reasons to believe that your classes are normally distributed, with equal variances, the logistic function is likely to be the best model for the class probabilities.

Answer 3

Let me provide some perspectives from epidemiology. In epidemiology, we are generally interested in risks, which is roughly equivalent to probability $p$. However, one important reason why we so often work with $\text{odds}=\frac{p}{1-p}$ is because of the case-control study design, in which we sample cases (diseased) and controls (non-diseased) independently and look at the proportion of the 2 samples being exposed to a risk factor. This is called a retrospective design in contrast to the prospective design where two samples with different risk factors are being compared for their risk for developing diseases.

Now the neat thing about the odds ratio is that it is invariant to both design -- the retrospective and prospective odds ratio are the same. This means one doesn't need to know the prevalence of the disease to estimate the odds ratio, unlike the risk ratio, from a case-control study.

More interestingly, this invariance of the odds ratio extends to logistic regression, such that we can analyse a retrospective study as if we were analysing a prospective study, under some assumptions (ref). This makes it a lot easier to analyse retrospective studies as we don't have to model the exposures $\boldsymbol{x}$ jointly.

Finally, although the odds ratio is difficult to interpret, with a rare disease, it approximates the risk ratio. That's another reason why modeling the odds is so common in epidemiology.

Answer 4

I think we can see the logistic regression from the perspective of Boltzmann distribution in physics/Gibbs distribution(also refer to this thread) or the log linear model in statistics.

We can treat the matrix(just view it from the softmax perspective), as the potentials between each visible feature variable and each hidden variable(the y's, if it is logistic regression there are two y's).

The $\theta^{(i)}$ is just the sum of the ith y and its potentials between all features, and the $e$ makes it the product.

And we can see that it can date back to 1868:

The Boltzmann distribution is named after Ludwig Boltzmann who first formulated it in 1868 during his studies of the statistical mechanics of gases in thermal equilibrium.