7
$\begingroup$

This issue may be a little trivial, but I really want to get some inspiration. I know the formula of logistic regression, but have no idea how it is deduced. How can I deduce this formula from more basic math theory? Is this the best function to fit binary classification?

$$\frac{p(x)}{1-p(x)}=e^{\beta_0+\beta_1 x}.$$

$\endgroup$
1
  • 3
    $\begingroup$ Use algebra, beginning with the logistic model. $\endgroup$
    – whuber
    Commented Nov 26, 2022 at 14:29

2 Answers 2

5
$\begingroup$

There are several ways how the logistic function can occur as an expression that is derived based on some underlying principle.

Derivation as Bayes factor

The formula occurs when one computes the Bayes factor where one assumes that the groups are normal distributed.

$$\frac{P(Y = 1 | X=x)}{P(Y = 0 | X=x)} = \frac{P(Y = 1)}{P(Y = 0)} \frac{\frac{1}{\sqrt{2\pi\sigma_1^2}}\text{exp}\left(-\frac{(x-\mu_1)^2}{2\sigma_1^2}\right)}{\frac{1}{\sqrt{2\pi\sigma_0^2}}\text{exp}\left(-\frac{(x-\mu_0)^2}{2\sigma_0^2}\right)} = \text{exp}\left(a + bx + cx^2 \right)$$

where $a$, $b$ and $c$ are functions of the prior ratio $P(Y = 1)/P(Y = 0)$ and of the means $\mu_1$, $\mu_0$ and of the standard deviations $\sigma_1$ and $\sigma_0$. If the standard deviations are the same then the quadratic term is zero.

Derivation as natural or canonical parameter

Logistic regression is often used for binary data and models the probability parameter $p$ for a Bernoulli distributed variable. For the binomial distribution the inverse of the logistic function is the logit function which is the natural parameter of the Bernoulli and binomial distribution.

If we write a distribution in the form of an exponential family

$$f(y|\theta) = h(y) \text{exp}\left(\eta(\theta) T(y) - A(\theta) \right)$$

Then we get for the Bernoulli distribution

$$\begin{array}{rcl} \theta &=& p\\ \eta(\theta) = \eta(p)& = &\text{log}\left( \frac{p}{1-p} \right)\\ T(y)& =& y\\ A(\theta) &= &-\text{log}(1-p)\\ h(y)& = &1 \end{array}$$

$$f(y|\theta) = \text{exp}\left(\text{log}\left( \frac{p}{1-p} \right) y + \text{log}(1-p) \right) = p^y (1-p)^{1-y}$$

and the logit function $\eta = \text{log}\left( \frac{p}{1-p} \right)$ is the natural parameter.

What is "natural" about the natural parameterization of an exponential family and the natural parameter space?

Relation to Boltzmann distribution and maximum entropy

The logistic function can originate from an exponential function for different outcomes

$$p(y;x) \propto \text{exp}(f(y,x))$$

and with the normalisation using the sum $\sum_{\forall y} p(y;x)$ it becomes a logistic function

$$p(y;x) = \frac{\text{exp}(f(y,x))}{\text{other stuff}+\text{exp}(f(y,x))}$$

The connection with entropy is in the exponential function $p(y;x) \propto \text{exp}(f(y,x))$ which is similar to the Boltzmann distribution.

See for more:

Logistic regression and maximum entropy

Maximum Entropy and Multinomial Logistic Function

Derivation as physical model

The logistic function also occurs as the solution of a differential equation

$$f^\prime(x) = f(x) \left(1-f(x)\right)$$

This occurs in growth models.

Latent variable model with logistic distribution.

Like probit regression a logistic regression can also be described with a latent variable model, replacing with a logistic distributed latent variable instead of Gaussian distributed latent variable.

It might not be so clear where a logistic distributed latent variable would come from (Confusion around logistic regression and the logistic distribution) and it might seem as an unreasonable idea, but there can be mechanisms for a logistic distributed latent variable as follows:

There is a relationship between the logistic distribution and the Gumbel distribution. If $X_i \sim Gumbel(\mu_i,\beta)$ then $X_2-X_1 \sim Logistic(\mu_2-\mu_1,\beta)$. So the probability for the event $X_1>X_2$ is related to the logistic distribution and it's CDF which is the logistic function. (similarly for more than $X_i$, the event that $X_1$ is the maximum can also be related to $X_1>max(X_2,\dots,X_n)$ and is also logistic distributed)

This situation occurs in discrete choice models. Another variant is a relationship for Cox proportional hazards models or other proportional probabilities based on exponential functions. If the probabilities for a specific class are proportional to an exponential function $P(Y = i) \propto e^{\alpha_i x}$ (like for instance the proportional hazards), then this is in normalized form $P(Y = i) = \frac{e^{\alpha_i x}}{\sum_{j=1}^n e^{\alpha_j x}}$ . In the bivariate case it reduces to a logistic function $$P(Y = 1) = \frac{e^{\alpha_1 x}}{e^{\alpha_1 x}+e^{\alpha_2 x}} = \frac{1}{1+e^{-(\alpha_1-\alpha_2) x}} $$

A derivation of the above based on a theory for population choice behaviour was made for the multinomial variant by McFadden in 1972 "Conditional logit analysis of qualitative choice behavior" and named the conditional logit model.

$\endgroup$
2
13
$\begingroup$

The logistic regression model is an instance of Generalized Linear Models (GLM) and arises as follows. Suppose we observe random variables that can take only zeros and ones, that is, let us have a sample $Y_1,\ldots, Y_n$, where $Y_i$ is a binary random variable. Suppose also that the elements of the sample are independent of each other, and each of them follows a Bernoulli distribution with success probability $p_i$.

Mathematically this can be expressed by

$$ Y_i\,\overset{\textrm{iid}}{\sim}\,\text{Ber}(p_i),\quad i=1,\ldots,n. $$

Now, for some good reason, you believe that the $p_i$s may not necessarily be equal and that they depend on some features that you observe or that you have control over. Let these features be $X_i = (X_{i1},\ldots, X_{ip})$, for $i=1,\ldots,n$. Suppose that a linear relationship between the probabilities $p_i$ and the $X_i$ is fine, say

$$ p_i = \psi(X_i^\top\beta),\tag{*} $$

where $\beta = (\beta_1,\ldots,\beta_p)$ are the unknown coefficients that determine such a relation and $\psi(x) = x$ is the identity function. However, you see right away that this representation is flawed because you have no guarantees that $X_i^\top \beta$ will be valid probabilities, i.e. will be in (0,1).

To overcome this issue, you have to use a suitable $\psi$ in (*) that guarantees this condition. One choice is to take $\psi(x) = e^{x}/(1+e^x)$ and the relation becomes

$$ p_i = \frac{e^{X_i\beta}}{1+e^{X_i\beta}}\, $$

or equivalently

$$ \text{logit}(p_i) = \log\left(\frac{p_i}{1-p_i}\right) = X_i\beta\,\tag{**} $$

The equation (**), or its equivalent version, is one of the reasons why this model is called logistic regression. By the way, this specific $\psi(x)$ is the cumulative distribution function of a standard logistic random variable. Other choices for $\psi$ are possible (i.e. the probit function which leads to the probit regression), although there are some special reasons for using the logit function.

This is meant as an intuitive explanation but if you need more mathematical details, please refer to (the bible of) GLM, i.e. McCullagh & Nelder (1983) Generalized Linear Models, 2nd edition, CRC Press, doi).

$\endgroup$
1
  • 3
    $\begingroup$ +1. For providing an outline and leaving rhe formal work for OP. $\endgroup$ Commented Nov 26, 2022 at 14:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.