Linear discriminant analysis and logistic regression

Question

I have found in the script of the Machine Learning lecture CS229 by Andrew Ng at Stanford University, that he claims that (at least in the case of only two classes) the posterior of the linear discriminant analysis (LDA) is also a logistic function. Given the scenario:

$y$ ~ $Bernoulli(\pi)$

$x|y=0$ ~ $N(\mu_0, \Sigma)$

$x|y=1$ ~ $N(\mu_1, \Sigma)$,

he states that if we consider $p(y=1|x;\pi, \Sigma, \mu_0, \mu_1)$ to be a function of $x$, we find:

$$p(y=1|x;\pi, \Sigma, \mu_0, \mu_1) = \frac{1}{1+\exp(-\theta^Tx)}$$ where $\theta$ is some appropriate function of $\pi, \Sigma, \mu_0, \mu_1$ (that he unfortunately does not state precisely). This, of course, is the definition of the logistic regression (LR) scenario as well. However, he states that the decision boundaries of LDA and LR generally will still look differently, when being trained on the same data set. My questions now are:

1.) Does this formula for the posterior only hold in the case of two classes, or can we write the posterior of LDA as a logistic function also for more classes?

2.) How does the function $\theta$ look like?

3.) How can LDA and LR have different decision boundaries, even if they have the same posterior function?

4.) How can we derive the logistic function from the LDA equations? So can someone sketch the rough derivation?

5.) I am a bit confused about the sentence that we have to consider "$p(y=1|x;\pi, \Sigma, \mu_0, \mu_1)$ to be a function of $x$" to derive the logistic function from LDA. Dont we always assume the posterior to be a function of $x$ anyway?

Answer 1

For the first question, you have to use Bayes' theorem. Let $\pi_0, \pi_1$ be prior probabilities on the two classes, and denote the multinormal density function by $f$. Then $$ \DeclareMathOperator{\P}{\mathbb{P}} \P(Y=1 \mid X=x) = \frac{\pi_1 \exp \left( -\frac12 (x-\mu_1)^T \Sigma^{-1}(x-\mu_1) \right)}{\pi_1 \exp \left( -\frac12 (x-\mu_1)^T \Sigma^{-1}(x-\mu_1) \right) + \pi_0 \exp \left( -\frac12 (x-\mu_0)^T \Sigma^{-1}(x-\mu_0) \right)} \\ = \frac1{1+\frac{\pi_0}{\pi_1} \exp\left( -\frac12 (x-\mu_0)^T \Sigma^{-1} (x-\mu_0) + \frac12 (x-\mu_1)^T\Sigma^{-1} (x-\mu_1)\right)} $$ and after some more algebra this finally simplifies to $$ \P(Y=1 \mid X=x) = \frac1{1+\exp\left( -\frac12 (x-\mu_0)^T \Sigma^{-1} (x-\mu_0) + \frac12 (x-\mu_1)^T\Sigma^{-1} (x-\mu_1)\right)+\ln(\pi_0/\pi_1)} $$ which has the logistic regression form. But this is somewhat different from the claim you attributes to Andrew Ng, as he seems to leave the prior probabilities out (or just assume both is $\frac12$.)

You can replicate this argument in the case of three or more classes.