1
$\begingroup$

I have a question concerning the evaluation of mixture models. Is there a gold standard to compute the goodness of a fit for a mixture model?

What I am concerned about is how one would evaluate if one, two or three gaussians fit a given distribution better. Truly, one could visually inspect that but I am looking for an automated way that has a statistical meaning.

My initial idea was to measure the KS statistic between the observed distribution and sampled distributions by the estimated mean and variance for each model. Admittedly, I am not an expert for mixture models so I might miss something obvious here.

So I guess, what I am looking for is some kind of likelihood ratio test than gives me the best performing model for one, two or three overlapping distributions.

I am very thankful for any keywords and links that I can look up!

$\endgroup$
0
$\begingroup$

You can use a model selection tool such as AIC or BIC to compare the models. However, this does not tell you about the goodness of fit. The same applies to the Likelihood ratio.

A formal goodness of fit test can be conducted by using the chi-square goodness of fit test. This is very sensitive to the choice of the bin-width, though.

A less formal, and more visual, goodness of fit test is the QQ-envelope which is obtained as follows:

  1. Fit the mixture model to your data.
  2. Simulate $N$ (large) samples from the fitted model of the same size as the original sample.
  3. Calculate the QQ plots for each simulated sample against the original sample, and plot all of them together. This will generate an envelope that tells you how good or bad your model reproduces the data.

You can use this tool to identify areas where the model produces a poor fit.

$\endgroup$
  • $\begingroup$ Right, so as far as I understand your second suggestion this is somehow similar to measure the KS distance. I wanted to do 1+2 as you proposed and measure the greatest distance between the cumulative density functions. However, I am more interested in the BIC/AIC method. With your answer and the answer above I found the following manuscript that might help for future references: cs.cmu.edu/~epxing/Class/10701-06f/project-reports/yu.pdf $\endgroup$ – Kam Sen Jul 23 '14 at 9:37
3
$\begingroup$

I am not sure there is a single gold standard but I know that the Bayesian Information Criterion (BIC) is used to evaluate the fit of a mixture model. The R Mclust package uses this as its default option in choosing the best fit among mixture models with different numbers of components. You can find a source for the math behind this but the BIC essentially computes the total log likelihood for the data given the model and penalizes it by the number of parameters in the model.

$\endgroup$
  • 1
    $\begingroup$ I am doing my stats for a while now in python. I found the following documentation for model selection in scikitlearn that seems to be able to compute the mixture model and can compute the BIC/AIC: scikit-learn.org/stable/auto_examples/mixture/… $\endgroup$ – Kam Sen Jul 23 '14 at 9:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.