8
$\begingroup$

I know that every class has the same covariance matrix $\Sigma$ in linear discriminant analysis (LDA), and in quadratic discriminant analysis (QDA) they are different. When using gaussian mixture model (GMM) in supervised classification, we fit a Gaussian with its own mean and variance to each class in data. So what is the difference between QDA and GMM?

Is my understanding of GMM wrong? Maybe I should fit more than one Gaussian to each class in order to model subgroups in it. But I am not sure if this is true or not.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Please don't forget, when asking a question, to expand your abbreviations. $\endgroup$ – ttnphns Mar 1 '16 at 8:03
  • 1
    $\begingroup$ A discriminant analysis is not only a classification but also a dimensionality reduction. In DA, classifiers are the discriminants extracted, not the input variables. $\endgroup$ – ttnphns Mar 1 '16 at 8:06
  • 1
    $\begingroup$ So, if we consider just the classification part, are QDA and GMM the same? $\endgroup$ – groove Mar 1 '16 at 9:04
5
$\begingroup$

If you're given class labels $c$ and fit a generative model $p(x, c) = p(c) p(x|c)$, and use the conditional distribution $p(c|x)$ for classification, then yes you're essentially performing QDA (the decision boundary will be quadratic in $x$). Under this generative model, the marginal distribution of the data $x$ is exactly the GMM density (say you have $K$ classes):

$$p(x) = \sum_{k \in \{1,...,K\}} p(c=k) p(x|c=k) = \sum_{k=1}^K \pi_k \mathcal{N}({x};{\mu}_k, {\Sigma}_k)$$

"Gaussian mixture" typically refers to the marginal distribution above, which is a distribution over $x$ alone, as we often don't have access to the class labels $c$.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.