5
$\begingroup$

I know that there are topics about this question but in my view, the answers are not clear enough. I don't understand the main difference between Linear Discriminant Analysis (LDA) and Gaussian Mixture Models (GMM).

Both have the same purpose : determine the posteriori $P(G=j|X=x)$, and maximize it for a certain $j$ in order to attribute class $j$ to $x$. I have the feeling that in GMM the way we estimate our parameters (EM algorithm) is the difference. Or, I don't know maybe the difference is that in LDA we want to draw an hyperplane in order to classify after any data ?
Because we agree that basically, LDA data correspond to a gaussian mixture model. It's just the way parameters are estimated that differs no?
Well as you can see, I'm a bit confused. I hope someone could explain me.

$\endgroup$
5
  • $\begingroup$ stats.stackexchange.com/questions/199275/… $\endgroup$
    – Taylor
    Jan 7, 2017 at 0:29
  • $\begingroup$ What does it mean? That GMM is an unsupervised learning algorithm and DA not, because we know in that case the labels? That is to say, in DA we have the class labels, and we build a model in order to classify new data. In GMM, we don't know the labels of our data, and we just want to classify them, knowing that they are gaussians? Thanks btw, for your answer. $\endgroup$
    – Mohamed
    Jan 7, 2017 at 1:21
  • 1
    $\begingroup$ DA is a dimensionality reduction technique for continuous variables, supervised by one categorical variable (i.e. alternative classes), and then classification in the reduced space to those classes. GMM is a latent categorical variable dimensionality reduction technique for continuous variables, it is unsupervised: no classes preexist: they are being extracted. GMM is closer to (model based) cluster analysis. $\endgroup$
    – ttnphns
    Jan 7, 2017 at 2:00
  • $\begingroup$ Thank you. I see. But why do we use EM algorithm in GMM and use basics estimators for our parameters in LDA? $\endgroup$
    – Mohamed
    Jan 7, 2017 at 2:27
  • $\begingroup$ Tell me if i'm wrong, but we use basic estimators because we are able to determine them in each class (because we have the labels). In GMM, we have to use a different method because we don't have the class labels right? $\endgroup$
    – Mohamed
    Jan 7, 2017 at 3:05

1 Answer 1

3
$\begingroup$

The building blocks of LDA and GMM are similar i.e both Gaussian but there are many differences. In GMM we are trying to estimate a distribution in the following form:
$ p(\boldsymbol{ x}|\theta) = \sum_{z=1}^K\pi_z \mathcal{N}(\boldsymbol{x|\tilde{\mu_z},\tilde{\Sigma_z}}) $

This is a density estimation problem, trying to estimate the density of an arbitrary distribution. The variable z is a hidden variable and the parameters $(\pi_z,\tilde{\mu_z},\tilde{\Sigma_z})$ are obtained via the EM algorithm. If you would like to do supervised classification for two classes you would train one model for each class $p(\boldsymbol{ x}|\theta_1)$ and $p(\boldsymbol{ x}|\theta_2)$ and select the model with the largest likelihood.

$ \hat{y}=\underset{y}{\operatorname{arg\,max}}\, p(\boldsymbol{ x}|\theta_y) $

THe LDA approaches the problem by assuming that the conditional probability density functions for each class $p(x|y=0)$ and $p(x|y=1)$ that are Multivariate normal distribution with mean and covariance parameters $( \mu_0, \Sigma_0)$ and $(\vec \mu_1, \Sigma_1)$. You would select a class as follows:

$ \hat{y}=\underset{y}{\operatorname{arg\,max}}\, p(y|x)=\underset{y}{\operatorname{arg\,max}}\, p(x|y)p(y) $

Where $p(y)$ is the prior. With some math one can show this is the same as:

$ (x- \mu_0)^T \Sigma_0^{-1} ( x- \vec \mu_0) + \ln|\Sigma_0| - ( x- \mu_1)^T \Sigma_1^{-1} ( x- \mu_1) - \ln|\Sigma_1| \ > \ T $

Where we predict points as being from the second class if the log of the likelihood ratios is below some threshold T.

$\endgroup$
1
  • 1
    $\begingroup$ So GMM is ML and LDA is MAP ? $\endgroup$
    – bicepjai
    Jul 27, 2017 at 19:27

This site is temporarily in read-only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .