1
$\begingroup$

I was studying the book "An Introduction to Statistical Learning" by Gareth James, Daniela Witten, et al. and saw the following line on page 151:

In the LDA framework, we can see from (4.12) to (4.13) (and a bit of simple algebra) that the log odds is given by:

$$ log \left(\frac{p_1(x)}{1-p_1(x)}\right) = log \left(\frac{p_1(x)}{p_2(x)}\right) = c_0 + c_1(x) $$

How can this proved?

$\endgroup$
2
  • $\begingroup$ It is worth pointing out that the book you are refering to the first edition of James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning. New York: Springer, and the pdf is freely available from statlearning.com. The first author is Gareth James rather than Trevor Hastie. $\endgroup$ Oct 24, 2021 at 6:26
  • $\begingroup$ Actually, James et al (2013) made a much stronger statement than that stated in your question. Instead of the general function $c_1(x)$ that you have written, James et claim that the right hand side is $c_0+c_1 x$ where $c_1$ is a function of $\mu_1$, $\mu_2$ and $\sigma^2$ but not of $x$. This difference is crucial because it shows that the log-odds is linear in $x$, which is what is required to make the LDA model equivalent to logistic regression. $\endgroup$ Oct 24, 2021 at 6:37

2 Answers 2

3
$\begingroup$

Its just algebra.

In the first edition of the book, equation 4.12 states that the posterior probability for each class can be written using Bayes rule as

$$ p_k = \dfrac{\pi_k f(x, \mu_k)}{\sum_k \pi_k f(x, \mu_k)} $$

Here, $\pi_k$ is the prior probability of class $k$ and $f(x, \mu_k)$ is a gaussian density with mean $\mu_k$. The density of $x$ for each class is assumed to have the same variance.

Now comes the algebra. Note

$$ \log \left( \dfrac{p_1}{p_2} \right) = \log\left( \dfrac{\pi_1 f(x, \mu_1)}{\pi_2 f(x, \mu_2)} \right) $$

Leveraging some properties of logs

$$ = \underbrace{\log(\pi_1/\pi_2)}_{c_0} + \underbrace{\log(f(x, \mu_1)/f(x, \mu_2))}_{c_1(x)}$$

EDIT: Gordon Smyth correctly notes that a much stronger statement can be made about $c(x)$, namely that it is linear in $x$. This is also rather straight forward to show.

Since LDA makes the assumption of homogeneity in the variance term between classes, $\log(f(x, \mu_1) / f(x, \mu_2))$ simplifies to

$$ \log(f(x, \mu_1) / f(x, \mu_2)) = -\dfrac{1}{2\sigma^2} \Big( (x-\mu_1)^2 - (x- \mu_2)^2\Big)$$

More algebra shows the $x^2$ term cancels out, yielding

$$ = - \dfrac{1}{2\sigma^2} ( -2 \mu_1 x + \mu_1^2 + 2\mu_2x - \mu_2^2) = \dfrac{1}{\sigma^2} \Big( (\mu_1 - \mu_2)\cdot x + 2(\mu_2^2 - \mu_1^2) \Big)$$

which is clearly a linear function in $x$. You could do some re-arranging so that $c_0$ has all the constant terms and $c(x)$ has only terms involving $x$, but it largely doesn't matter.

$\endgroup$
1
  • $\begingroup$ James et al (2013) actually claimed that $\log (p_1/p_2) = c_0 + c_1 x$ where a $c_1$ is constant not depending on $x$. To prove their claim you also need to show that $c_1(x)$ is linear in $x$. $\endgroup$ Oct 24, 2021 at 6:18
1
$\begingroup$

You can view the logistic function as originating from two normal distributed samples.

Say $U \sim N(\mu_U , \sigma)$ and $V \sim N(\mu_Y, \sigma)$ then the relative density of $U$ and $V$ as function of the position is

$$\frac{f_U(x)}{f_V(x)} = \frac{e^{\frac{(x-\mu_U)^2}{2\sigma^2}}}{e^{\frac{(x-\mu_V)^2}{2\sigma^2}}} = e^{\frac{(x-\mu_U)^2 - (x-\mu_V)^2}{2\sigma^2}} = e^{\frac{\mu_U^2+\mu_V^2 - 2x(\mu_U-\mu_V)}{2\sigma^2}} = e^{a+bx}$$

with $a= \frac{\mu_U^2+\mu_V^2}{2\sigma}$ and $b= \frac{\mu_U-\mu_V}{\sigma}$


With LDA you get these two normal distributed samples. LDA assumes that the samples are normal distributed and with the same covariance matrix. In the multidimensional picture you need an extra step but you get that the density ratio along the factor/component (the linear combination found with LDA) is just like the one dimensional picture above. I'll have to make a good illustration of this but you can see this by the points with equal odds ratios being on straight lines/planes (in particular the classifier for odds ratio equal to one is known).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.