I have been training a GMM (Gaussian Mixture, clustering / unsupervised) on two version of the same dataset: one training with all its features and one training after a PCA truncated to its 2 first principal components. Then I have been plotting their respective log-likelihood, given by .score() in scikit-learn api, against the number of clusters.

enter image description here enter image description here

I struggle to understand why the axis are not of the same range. I mean, why does the PCA model give me so small values (around -4) compared to the full model which gives values around 80?

I have actually no guess if the log-likelihood should lie in a particular range since we are dealing with probabilities.



1 Answer 1


(log-) likelihood of a mixture model

You have a model $g_{\theta}$ to describe some data sample $\mathbf{x}$, in this case your mixture model. This model is dependent on it's parameters, in this case the means, variances, and weights of the mixture components . For simplicity's sake we gather them in $\theta$.

The likelihood function gives us "the probability of the observed data under the model $g_{\theta}$. [...]. We think of (the likelihood function) as a function of $\theta$, with our data $\mathbf{x}$ fixed. " (1). That is, the value of the likelihood function tells us in some sense how 'plausible' the model at hand is, given the data that we have. As a value on it's own, this isn't very helpful to us - but it allows us to compare models with different parameter values. That's why it is useful, for example in model inference.

The log-likelihood function then is $$\ell(\theta|\mathbf{x})=\sum_{i=1}^{N} \log( g_\theta(x_i))$$

with in your case of a gaussian mixture model $g_\theta(x_i)$ being the density estimate $f(x_i)$

$$ f(x_i)= \sum_{m=1}^{M} \alpha_m \phi(x_i|\mu_m,\Sigma_m) $$

and the log-likelihood is

$$\ell(\theta|\mathbf{x})=\sum_{i=1}^{N} \log\left( \sum_{m=1}^{M} \alpha_m \phi(x_i|\mu_m,\Sigma_m)\right)$$

given the parameters $\theta=\{\alpha_1,...,\alpha_M,\mu_1,...,\mu_M,\Sigma_1,...,\Sigma_M\}$

Why the log-likelihood values in your example are different

We have seen by now, that the parameters as well as the data affect the values of the log-likelihood.
The models in your first image are trained on the original data set, while those in the second picture are trained on the results of the PCA you performed. As such the dimensionality of the two data sets and probably also the ranges of the features are different.
As you might know, the model parameters of your mixture models are determined via EM based on the data. I hope you see where this is going.

In the case you described, the data and by that also the parameters of models are different - comparing the values of the log-likelihood in your example won't give you the insight you might have looked for.


(1) Friedman, J., Hastie, T., & Tibshirani, R. (2001). The elements of statistical learning (Vol. 1, No. 10). New York: Springer series in statistics.

  • $\begingroup$ Many thanks, very detailed. Appreciate the reference too! $\endgroup$
    – Sylvain M
    Apr 9, 2019 at 14:07
  • $\begingroup$ You're welcome. Did that answer your question or is something left unclear? $\endgroup$
    – deemel
    Apr 9, 2019 at 14:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.