In CS229, Prof.Andrew Ng says( but doesn't prove ) the relation between Gaussian Discriminant Analysis and Logistic Regression (hypothesis function is sigmoid in both models). Is this (similarity of hypothesis function) because of y being distributed Bernoulli?

  • $\begingroup$ if possible, please provide the complete derivation $\endgroup$
    – euler16
    Commented Dec 12, 2016 at 11:03
  • $\begingroup$ derivation the parameters of max likelihood phi, sigma, u1 and u0 $\endgroup$
    – euler16
    Commented Dec 12, 2016 at 13:05
  • 1
    $\begingroup$ See stats.stackexchange.com/questions/188416/… $\endgroup$ Commented May 18, 2019 at 9:47

1 Answer 1


I'm not sure what it means that GDA "leads to the sigmoid hypothesis" but perhaps it refers to the relationship between GDA (aka Linear Discriminant Analysis, LDA) and logistic regression.

Before presenting the derivation, a few notes:

  • There are strong assumptions that GDA makes namely that the data being generated is from a mixture of gaussians with different means, $\mu_i$ and similar covariances, $\Sigma$. This latter assumption gives us the "Linear" part in LDA.
  • If the marginal distribution of the data is indeed Gaussian, then LDA performs much better than logistic regression (e.g. you will need 30% more data for a logistic regression to fit as well as an LDA model if the data, Efron 1975).
  • However logistic regression is more robust to deviations from normality, i.e. outliers.
  • LDA requires fitting a joint likelihood, whereas logistic regression only requires fitting a conditional likelihood.

As for deriving the relationship for the binary case (class 1/0), we begin by writing out the probability of a data point, $Y$, being generated from class 1:

$$Y=1 \rightarrow X \sim N(\mu_1, \Sigma)$$ $$Y=0 \rightarrow X \sim N(\mu_0, \Sigma)$$ $$P(Y=1) = \pi_1$$ $$P(Y=0) = \pi_0$$

Using Bayes Theorem:

$$P(Y = 1|X=x) = \frac{\pi_1P(X=x|Y=1)}{\pi_0P(X=x|Y=0) + \pi_1P(X=x|Y=1)}$$

$$ = \frac{\pi_1 e^{-(x - \mu_1)^T\Sigma^{-1}(x - \mu_1)/2}}{\pi_0e^{-(x - \mu_0)^T\Sigma^{-1}(x - \mu_0)/2} + \pi_1e^{-(x - \mu_1)^T\Sigma^{-1}(x - \mu_1)/2}}$$

Now we can write the odds in favor of class 1:

$$\frac{P(Y = 1|X=x)}{P(Y = 0|X=x)} = \frac{\pi_1}{\pi_0}e^{-(x - \mu_1)^T\Sigma^{-1}(x - \mu_1)/2 + (x - \mu_0)^T\Sigma^{-1}(x - \mu_0)/2 }$$

And finally the log odds in favor of class 1:

$$log\frac{P(Y = 1|X=x)}{P(Y = 0|X=x)} = log(\frac{\pi_1}{\pi_0}) -(x - \mu_1)^T\Sigma^{-1}(x - \mu_1)/2 + (x - \mu_0)^T\Sigma^{-1}(x - \mu_0)/2 $$

$$ = log(\frac{\pi_1}{\pi_0}) + (\mu_1 - \mu_0)^T\Sigma^{-1}x + \frac{1}{2}(\mu_0^T\Sigma^{-1}\mu_0 - \mu_1^T\Sigma^{-1}\mu_1)$$

which is linear in $x$ and can be written in the form:

$$logit P(Y=1|X=x) = \alpha + x^T\beta$$

where $\alpha = log(\frac{\pi_1}{\pi_0}) + \frac{1}{2}(\mu_0^T\Sigma^{-1}\mu_0 - \mu_1^T\Sigma^{-1}\mu_1)$ and
$\beta = \Sigma^{-1}(\mu_1 - \mu_0)$

This shows the equivalence with logistic regression.

  • $\begingroup$ Thank you @ilanman . This was exactly what I was asking for. Sorry for the poor question framing $\endgroup$
    – euler16
    Commented Dec 13, 2016 at 12:52
  • $\begingroup$ Glad it helped! Perhaps you can reword your question so they open it up again? $\endgroup$
    – ilanman
    Commented Dec 13, 2016 at 13:18

Not the answer you're looking for? Browse other questions tagged or ask your own question.