1
$\begingroup$

There seems to be (at least) two ways that one could evaluate the fit of a Gaussian Mixture Model (GMM) to a data set.

First, a probabilistic score, is the log likelihood of a set of points $D$ under the GMM:

$$ \log p(D|\theta) = \sum_{i=1}^{N_{points}} \log f(x_i) $$

where I've defined the GMM probability $f(x_i)=\sum_{k=1}^{N_{components}} \pi_k \mathcal{N}(x_i|\mu_k,\Sigma_k)$. Consider the case that the data are binned, in which we can use the bin center approximation: $$ \log p(D|\theta) = \sum_{i=1}^{N_{bins}} |b_i|\log (f(b_i)\Delta b_i) $$

where $b_i$ is the position of the bin center, |$b_i$| is the number of observations, and $\Delta b_i$ is the bin width.

Second, a density score. Since the binned data is also a probability density we can use density metrics like the $L^2$ norm: $$ \begin{align} L^2(f,D) &= \int_x \left(f(x) - D(x)\right)^2dx \\&= \int_x f^2(x)dx -2\int_x D(x)f(x)dx + \int_xD^2(x)dx \end{align} $$ The first term is easy to compute because it's the sum of integrals over all products of components of $f$, so that's analytic. The last term is just the sum over the squared values of the histogram. The middle term, again using the bin center approximation, is nearly identical to the log-likelihood: $$ \int_x D(x)f(x) = \sum_{i=1}^{N_{bins}}|b_i| f(b_i)\Delta b_i $$ Here are my questions: what are the differences between these "scores"? When should you each of them, or alternatively other scores like the correlation? The appeal (to me) of something like the $L^2$ is that one could compare, using the same metric, multiple GMMs to each other or to $D$ using the same function. On the other hand, the probabilistic score appears to be weighted by the number of observations.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.