1
$\begingroup$

I understand that binary logistic regression is applied to binary classification problems where the dependent variable $Y$ has only two possible outcomes. The independent variables are $x$. The result of logistic regression is assigning a probability $p$ to one of the two outcomes and a probability $p-1$ to the other possible outcome.

I am confused on how the linear combination of the independent variables $w_1 x_1 +w_2 x_2 +w_3 x_3 $, $log \frac {p} {1-p}$, the probability $p$, the logistic function $\frac {1}{1+e^{-x}}$ are connected to each other.

Can someone help me logically understand how these concepts go together so I can finally appreciate how logistic regression works?

Thank you!

$\endgroup$
2

1 Answer 1

2
$\begingroup$

This is the logistic regression model, where the log-odds are posited to change as a linear function of some predictors.

$$ \log\bigg( \dfrac{p}{1-p} \bigg) = X\beta $$

$X\beta$ is the linear combination. You denote it as $w_1 x_1 +w_2 x_2 +w_3 x_3 $. A more traditional way to write it would use $\beta$ as the symbol for coefficients and would involve an intercept, so more like: $$X\beta = \beta_0 +\beta1x_1 + \beta_2x_2+\beta_3x_3$$

In order to solve for $p$, we must do some algebra.

$$ \log\bigg( \dfrac{p}{1-p} \bigg) = X\beta\implies\\ \dfrac{p}{1-p} = \exp(X\beta)\implies\\ p = (1 - p) \exp(X\beta)\implies\\ p = \exp(X\beta) - p \exp(X\beta)\implies\\ p+p\exp(X\beta) = \exp(X\beta)\implies\\ p(1 + \exp(X\beta)) = \exp(X\beta)\implies\\ p = \dfrac{\exp(X\beta)}{1 + \exp(X\beta)\implies}\\ p = \bigg( \dfrac{1 + \exp(X\beta)}{\exp(X\beta)} \bigg)^{-1}\implies\\ p =\bigg( \dfrac{1}{\exp(X\beta)} + 1 \bigg)^{-1}\implies\\ p =\bigg( \exp(-X\beta) + 1 \bigg)^{-1}\implies\\ p = \dfrac{1}{1 + \exp(-X\beta)} $$

$\endgroup$
2
  • $\begingroup$ Great help. The starting point, as you mention, is that "the log-odds are posited to change as a linear function of some predictors". I follow the derivation. But, naively, what the purpose of take the log of the odds and setting that equal to the linear combination of the independent variables? $\endgroup$ Dec 15, 2021 at 2:13
  • $\begingroup$ That’s called the “link function” of the generalized linear model, and you can use other link functions with Binomial $y_i$ variables. Probit regression uses the standard normal quantile (inverse CDF) function, and other inverse CDFs are viable, too. $\endgroup$
    – Dave
    Dec 15, 2021 at 2:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.