7
$\begingroup$

This issue may be a little trivial, but I really want to get some inspiration. I know the formula of logistic regression, but have no idea how it is deduced. How can I deduce this formula from more basic math theory? Is this the best function to fit binary classification?

$$\frac{p(x)}{1-p(x)}=e^{\beta_0+\beta_1 x}.$$

$\endgroup$
1
  • 3
    $\begingroup$ Use algebra, beginning with the logistic model. $\endgroup$
    – whuber
    Nov 26, 2022 at 14:29

2 Answers 2

4
$\begingroup$

There are several ways how the logistic function can occur as an expression that is derived based on some underlying principle.

Derivation as Bayes factor

The formula occurs when one computes the Bayes factor where one assumes that the groups are normal distributed.

$$\frac{P(Y = 1 | X=x)}{P(Y = 0 | X=x)} = \frac{P(Y = 1)}{P(Y = 0)} \frac{\frac{1}{\sqrt{2\pi\sigma_1^2}}\text{exp}\left(-\frac{(x-\mu_1)^2}{2\sigma_1^2}\right)}{\frac{1}{\sqrt{2\pi\sigma_0^2}}\text{exp}\left(-\frac{(x-\mu_0)^2}{2\sigma_0^2}\right)} = \text{exp}\left(a + bx + cx^2 \right)$$

where $a$, $b$ and $c$ are functions of the prior ratio $P(Y = 1)/P(Y = 0)$ and of the means $\mu_1$, $\mu_0$ and of the standard deviations $\sigma_1$ and $\sigma_0$. If the standard deviations are the same then the quadratic term is zero.

Derivation as natural or canonical parameter

Logistic regression is often used for binary data and models the probability parameter $p$ for a Bernoulli distributed variable. For the binomial distribution the inverse of the logistic function is the logit function which is the natural parameter of the Bernoulli and binomial distribution.

If we write a distribution in the form of an exponential family

$$f(y|\theta) = h(y) \text{exp}\left(\eta(\theta) T(y) - A(\theta) \right)$$

Then we get for the Bernoulli distribution

$$\begin{array}{rcl} \theta &=& p\\ \eta(\theta) = \eta(p)& = &\text{log}\left( \frac{p}{1-p} \right)\\ T(y)& =& y\\ A(\theta) &= &-\text{log}(1-p)\\ h(y)& = &1 \end{array}$$

$$f(y|\theta) = \text{exp}\left(\text{log}\left( \frac{p}{1-p} \right) y + \text{log}(1-p) \right) = p^y (1-p)^{1-y}$$

and the logit function $\eta = \text{log}\left( \frac{p}{1-p} \right)$ is the natural parameter.

What is "natural" about the natural parameterization of an exponential family and the natural parameter space?

Relation to Boltzmann distribution and maximum entropy

The logistic function can originate from an exponential function for different outcomes

$$p(y;x) \propto \text{exp}(f(y,x))$$

and with the normalisation using the sum $\sum_{\forall y} p(y;x)$ it becomes a logistic function

$$p(y;x) = \frac{\text{exp}(f(y,x))}{\text{other stuff}+\text{exp}(f(y,x))}$$

The connection with entropy is in the exponential function $p(y;x) \propto \text{exp}(f(y,x))$ which is similar to the Boltzmann distribution.

See for more:

Logistic regression and maximum entropy

Maximum Entropy and Multinomial Logistic Function

Derivation as physical model

The logistic function also occurs as the solution of a differential equation

$$f^\prime(x) = f(x) \left(1-f(x)\right)$$

This occurs in growth models.

$\endgroup$
2
13
$\begingroup$

The logistic regression model is an instance of Generalized Linear Models (GLM) and arises as follows. Suppose we observe random variables that can take only zeros and ones, that is, let us have a sample $Y_1,\ldots, Y_n$, where $Y_i$ is a binary random variable. Suppose also that the elements of the sample are independent of each other, and each of them follows a Bernoulli distribution with success probability $p_i$.

Mathematically this can be expressed by

$$ Y_i\,\overset{\textrm{iid}}{\sim}\,\text{Ber}(p_i),\quad i=1,\ldots,n. $$

Now, for some good reason, you believe that the $p_i$s may not necessarily be equal and that they depend on some features that you observe or that you have control over. Let these features be $X_i = (X_{i1},\ldots, X_{ip})$, for $i=1,\ldots,n$. Suppose that a linear relationship between the probabilities $p_i$ and the $X_i$ is fine, say

$$ p_i = \psi(X_i^\top\beta),\tag{*} $$

where $\beta = (\beta_1,\ldots,\beta_p)$ are the unknown coefficients that determine such a relation and $\psi(x) = x$ is the identity function. However, you see right away that this representation is flawed because you have no guarantees that $X_i^\top \beta$ will be valid probabilities, i.e. will be in (0,1).

To overcome this issue, you have to use a suitable $\psi$ in (*) that guarantees this condition. One choice is to take $\psi(x) = e^{x}/(1+e^x)$ and the relation becomes

$$ p_i = \frac{e^{X_i\beta}}{1+e^{X_i\beta}}\, $$

or equivalently

$$ \text{logit}(p_i) = \log\left(\frac{p_i}{1-p_i}\right) = X_i\beta\,\tag{**} $$

The equation (**), or its equivalent version, is one of the reasons why this model is called logistic regression. By the way, this specific $\psi(x)$ is the cumulative distribution function of a standard logistic random variable. Other choices for $\psi$ are possible (i.e. the probit function which leads to the probit regression), although there are some special reasons for using the logit function.

This is meant as an intuitive explanation but if you need more mathematical details, please refer to (the bible of) GLM, i.e. McCullagh & Nelder (1983) Generalized Linear Models, 2nd edition, CRC Press, doi).

$\endgroup$
1
  • 3
    $\begingroup$ +1. For providing an outline and leaving rhe formal work for OP. $\endgroup$ Nov 26, 2022 at 14:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.