Gaussian Discriminant Analysis and sigmoid function [closed]

Question

In CS229, Prof.Andrew Ng says( but doesn't prove ) the relation between Gaussian Discriminant Analysis and Logistic Regression (hypothesis function is sigmoid in both models). Is this (similarity of hypothesis function) because of y being distributed Bernoulli?

Answer 1

I'm not sure what it means that GDA "leads to the sigmoid hypothesis" but perhaps it refers to the relationship between GDA (aka Linear Discriminant Analysis, LDA) and logistic regression.

Before presenting the derivation, a few notes:

• There are strong assumptions that GDA makes namely that the data being generated is from a mixture of gaussians with different means, $\mu_i$ and similar covariances, $\Sigma$. This latter assumption gives us the "Linear" part in LDA.
• If the marginal distribution of the data is indeed Gaussian, then LDA performs much better than logistic regression (e.g. you will need 30% more data for a logistic regression to fit as well as an LDA model if the data, Efron 1975).
• However logistic regression is more robust to deviations from normality, i.e. outliers.
• LDA requires fitting a joint likelihood, whereas logistic regression only requires fitting a conditional likelihood.

As for deriving the relationship for the binary case (class 1/0), we begin by writing out the probability of a data point, $Y$, being generated from class 1:

$$Y=1 \rightarrow X \sim N(\mu_1, \Sigma)$$ $$Y=0 \rightarrow X \sim N(\mu_0, \Sigma)$$ $$P(Y=1) = \pi_1$$ $$P(Y=0) = \pi_0$$

Using Bayes Theorem:

$$P(Y = 1|X=x) = \frac{\pi_1P(X=x|Y=1)}{\pi_0P(X=x|Y=0) + \pi_1P(X=x|Y=1)}$$

$$ = \frac{\pi_1 e^{-(x - \mu_1)^T\Sigma^{-1}(x - \mu_1)/2}}{\pi_0e^{-(x - \mu_0)^T\Sigma^{-1}(x - \mu_0)/2} + \pi_1e^{-(x - \mu_1)^T\Sigma^{-1}(x - \mu_1)/2}}$$

Now we can write the odds in favor of class 1:

$$\frac{P(Y = 1|X=x)}{P(Y = 0|X=x)} = \frac{\pi_1}{\pi_0}e^{-(x - \mu_1)^T\Sigma^{-1}(x - \mu_1)/2 + (x - \mu_0)^T\Sigma^{-1}(x - \mu_0)/2 }$$

And finally the log odds in favor of class 1:

$$log\frac{P(Y = 1|X=x)}{P(Y = 0|X=x)} = log(\frac{\pi_1}{\pi_0}) -(x - \mu_1)^T\Sigma^{-1}(x - \mu_1)/2 + (x - \mu_0)^T\Sigma^{-1}(x - \mu_0)/2 $$

$$ = log(\frac{\pi_1}{\pi_0}) + (\mu_1 - \mu_0)^T\Sigma^{-1}x + \frac{1}{2}(\mu_0^T\Sigma^{-1}\mu_0 - \mu_1^T\Sigma^{-1}\mu_1)$$

which is linear in $x$ and can be written in the form:

$$logit P(Y=1|X=x) = \alpha + x^T\beta$$

where $\alpha = log(\frac{\pi_1}{\pi_0}) + \frac{1}{2}(\mu_0^T\Sigma^{-1}\mu_0 - \mu_1^T\Sigma^{-1}\mu_1)$ and
$\beta = \Sigma^{-1}(\mu_1 - \mu_0)$

This shows the equivalence with logistic regression.