1
$\begingroup$

In CS229, Prof.Andrew Ng says( but doesn't prove ) the relation between Gaussian Discriminant Analysis and Logistic Regression (hypothesis function is sigmoid in both models). Is this (similarity of hypothesis function) because of y being distributed Bernoulli?

$\endgroup$

closed as unclear what you're asking by kjetil b halvorsen, mdewey, Nick Cox, gung, whuber Dec 12 '16 at 14:53

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

6
$\begingroup$

I'm not sure what it means that GDA "leads to the sigmoid hypothesis" but perhaps it refers to the relationship between GDA (aka Linear Discriminant Analysis, LDA) and logistic regression.

Before presenting the derivation, a few notes:

  • There are strong assumptions that GDA makes namely that the data being generated is from a mixture of gaussians with different means, $\mu_i$ and similar covariances, $\Sigma$. This latter assumption gives us the "Linear" part in LDA.
  • If the marginal distribution of the data is indeed Gaussian, then LDA performs much better than logistic regression (e.g. you will need 30% more data for a logistic regression to fit as well as an LDA model if the data, Efron 1975).
  • However logistic regression is more robust to deviations from normality, i.e. outliers.
  • LDA requires fitting a joint likelihood, whereas logistic regression only requires fitting a conditional likelihood.

As for deriving the relationship for the binary case (class 1/0), we begin by writing out the probability of a data point, $Y$, being generated from class 1:

$$Y=1 \rightarrow X \sim N(\mu_1, \Sigma)$$ $$Y=0 \rightarrow X \sim N(\mu_0, \Sigma)$$ $$P(Y=1) = \pi_1$$ $$P(Y=0) = \pi_0$$

Using Bayes Theorem:

$$P(Y = 1|X=x) = \frac{\pi_1P(X=x|Y=1)}{\pi_0P(X=x|Y=0) + \pi_1P(X=x|Y=1)}$$

$$ = \frac{\pi_1 e^{-(x - \mu_1)^T\Sigma^{-1}(x - \mu_1)/2}}{\pi_0e^{-(x - \mu_0)^T\Sigma^{-1}(x - \mu_0)/2} + \pi_1e^{-(x - \mu_1)^T\Sigma^{-1}(x - \mu_1)/2}}$$

Now we can write the odds in favor of class 1:

$$\frac{P(Y = 1|X=x)}{P(Y = 0|X=x)} = \frac{\pi_1}{\pi_0}e^{-(x - \mu_1)^T\Sigma^{-1}(x - \mu_1)/2 + (x - \mu_0)^T\Sigma^{-1}(x - \mu_0)/2 }$$

And finally the log odds in favor of class 1:

$$log\frac{P(Y = 1|X=x)}{P(Y = 0|X=x)} = log(\frac{\pi_1}{\pi_0}) -(x - \mu_1)^T\Sigma^{-1}(x - \mu_1)/2 + (x - \mu_0)^T\Sigma^{-1}(x - \mu_0)/2 $$

$$ = log(\frac{\pi_1}{\pi_0}) + (\mu_1 - \mu_0)^T\Sigma^{-1}x + \frac{1}{2}(\mu_0^T\Sigma^{-1}\mu_0 - \mu_1^T\Sigma^{-1}\mu_1)$$

which is linear in $x$ and can be written in the form:

$$logit P(Y=1|X=x) = \alpha + x^T\beta$$

where $\alpha = log(\frac{\pi_1}{\pi_0}) + \frac{1}{2}(\mu_0^T\Sigma^{-1}\mu_0 - \mu_1^T\Sigma^{-1}\mu_1)$ and
$\beta = \Sigma^{-1}(\mu_1 - \mu_0)$

This shows the equivalence with logistic regression.

$\endgroup$
  • $\begingroup$ Thank you @ilanman . This was exactly what I was asking for. Sorry for the poor question framing $\endgroup$ – euler16 Dec 13 '16 at 12:52
  • $\begingroup$ Glad it helped! Perhaps you can reword your question so they open it up again? $\endgroup$ – ilanman Dec 13 '16 at 13:18

Not the answer you're looking for? Browse other questions tagged or ask your own question.