9
$\begingroup$

I first learned the logistic function in machine learning course, where it is just a function that map a real number to 0 to 1. We can use calculus to get its derivative and use the derivative for some optimization tasks.

Later, I learned it in statistic literature where there are log odds and bunch of probabilistic interpretations.

Today I am reviewing some differential equation literature, and found the logistic function is a solution for the following system:

$y'=y(1-y)$, and $y(0)=0.5$.


So, my question is more like a "chicken and egg" problem: where does the logistic function come from? Do we first have the function or the ODE?

$\endgroup$
  • 2
    $\begingroup$ It arises in many contexts. One relatively modern answer is that it's proportional to the folded logarithm. $\endgroup$ – whuber Apr 8 at 14:11
  • $\begingroup$ As a Riccati equation, its history in pure mathematics appears to exceed its history in applications. (Different answers here underline that specifying first dates is hazardous.) . $\endgroup$ – Nick Cox Apr 21 at 14:58
  • 2
    $\begingroup$ Wikipedia traces it to the 1840's at en.wikipedia.org/wiki/Logistic_function#History, but I'm confident Euler would have studied this function. $\endgroup$ – whuber May 19 at 21:45
8
$\begingroup$

I read from Strogatz's book that it was originated from modeling the human populations by Verhulst in 1838. Assume the population size is $N(t)$, then the per capita growth rate is $\dot N(t)/N(t)$. By assuming the per capita growth rate descreases linearly with the population size, we can have the logistic equation of following form: $$\dot N(t)=rN(1-\frac{N}{K}),$$ where $K$ is carrying capacity of the environment. From the equation, we can see that when $N$ is very small, the population grows approximately exponentially. As $N$ grows until half of the capacity, the derivative $\dot N$ is still increasing but slows down. Once $N$ passes the half line, the derivative decreases so we can see a bending curve (Interestingly, we can see this trend very roughly from the bended curves of cumulative Coronavirus cases) and the population asymptotically approaches the capacity. Up to this point, we observe the properties of a logistic function. Intuitively, when $N$ is larger the capacity, the population decreases. By further mathematical simplification of the equation, we have an equation like this: $$\frac{df(x)}{dx}=f(x)(1-f(x)),$$ which has analytical solotion as: $$f(x)=\frac{e^x}{e^x + C}.$$ With $C=1$ we have the logistic function.

| cite | improve this answer | |
$\endgroup$
3
$\begingroup$

I don't know about its history, but logistic function has a property which makes it attractive for machine learning and logistic regression:

If you have two normally distributed classes with equal variances, then the posterior probability of an observation to belong to one of these classes is given by the logistic function.

First, for any two classes $A$ and $B$ it follows from the Bayesian formula:

$$ P(B | x) = \frac{P(x | B) P(B)}{P(x)} = \frac{P(x | B) P(B)}{P(x | A) P(A) + P(x | B) P(B)} = \frac{1}{1 + \frac{P(x | A)P(A)} {P(x | B)P(B)}}. $$

If $x$ is continuous, so that the classes can be described by their PDFs, $f_A(x)$ and $f_B(x)$, the fraction $P(x | A) / P(x | B)$ can be expressed as:

$$ \frac{P(x | A)} {P(x | B)} = \lim_{\Delta x \rightarrow 0} \frac{f_A(x) \Delta x}{f_B(x) \Delta x} = \frac{f_A(x)}{f_B(x)}. $$

If the two classes are normally distributed, with equal variances:

$$ f_A(x) = \frac{1}{\sqrt{2 \pi} \sigma} \exp \left( -\frac{(x - \mu_A)^2} {2 \sigma^2} \right), ~ ~ ~ ~ ~ ~ ~ ~ ~ f_B(x) = \frac{1}{\sqrt{2 \pi} \sigma} \exp \left( -\frac{(x - \mu_B)^2} {2 \sigma^2} \right) $$

then the fraction $f_A(x) / f_B(x)$ can be written as:

$$ \frac{f_A(x)}{f_B(x)} = \exp \left( - \frac{(x - \mu_A)^2} {2 \sigma^2} + \frac{(x - \mu_B)^2} {2 \sigma^2} \right) = \exp \left( \frac{\mu_B^2 - \mu_A^2} {2 \sigma^2} + \frac{\mu_A - \mu_B} {\sigma^2} x \right), $$

and the whole term

$$ \frac{f_A(x)P(A)}{f_B(x)P(B)} = \exp \left( \ln \frac{P(A)}{P(B)} + \frac{\mu_B^2 - \mu_A^2} {2 \sigma^2} + \frac{\mu_A - \mu_B} {\sigma^2} x \right). $$

Denoting

$$ \beta_0 = \frac{\mu_A^2 - \mu_B^2} {2 \sigma^2} - \ln \frac{P(A)}{P(B)} ~ ~ ~ ~ ~ ~ ~ ~ \text{and} ~ ~ ~ ~ ~ ~ ~ ~  \beta_1 = \frac{\mu_B - \mu_A} {\sigma^2} $$

leads to the form commonly used in logistic regression:

$$ P(B | x) = \frac{1}{1 + \exp \left(-\beta_0 - \beta_1 x \right) }. $$

So, if you have reasons to believe that your classes are normally distributed, with equal variances, the logistic function is likely to be the best model for the class probabilities.

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

Let me provide some perspectives from epidemiology. In epidemiology, we are generally interested in risks, which is roughly equivalent to probability $p$. However, one important reason why we so often work with $\text{odds}=\frac{p}{1-p}$ is because of the case-control study design, in which we sample cases (diseased) and controls (non-diseased) independently and look at the proportion of the 2 samples being exposed to a risk factor. This is called a retrospective design in contrast to the prospective design where two samples with different risk factors are being compared for their risk for developing diseases.

Now the neat thing about the odds ratio is that it is invariant to both design -- the retrospective and prospective odds ratio are the same. This means one doesn't need to know the prevalence of the disease to estimate the odds ratio, unlike the risk ratio, from a case-control study.

More interestingly, this invariance of the odds ratio extends to logistic regression, such that we can analyse a retrospective study as if we were analysing a prospective study, under some assumptions (ref). This makes it a lot easier to analyse retrospective studies as we don't have to model the exposures $\boldsymbol{x}$ jointly.

Finally, although the odds ratio is difficult to interpret, with a rare disease, it approximates the risk ratio. That's another reason why modeling the odds is so common in epidemiology.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

I think we can see the logistic regression from the perspective of Boltzmann distribution in physics/Gibbs distribution(also refer to this thread) or the log linear model in statistics.

We can treat the matrix(just view it from the softmax perspective), as the potentials between each visible feature variable and each hidden variable(the y's, if it is logistic regression there are two y's).

enter image description here

The $\theta^{(i)}$ is just the sum of the ith y and its potentials between all features, and the $e$ makes it the product.

And we can see that it can date back to 1868:

The Boltzmann distribution is named after Ludwig Boltzmann who first formulated it in 1868 during his studies of the statistical mechanics of gases in thermal equilibrium.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.