11
$\begingroup$

Suppose I have a mixture of finitely many Gaussians with known weights, means, and standard deviations. The means are not equal. The mean and standard deviation of the mixture can be calculated, of course, since the moments are weighted averages of the moments of the components. The mixture is not a normal distribution, but how far from normal is it?

Mixture of Gaussians separated by 2 standard deviations vs. Gaussian with same mean and variance

The above image shows the probability densities for a Gaussian mixture with component means separated by $2$ standard deviations (of the components) and a single Gaussian with the same mean and variance.

Mixture of Gaussians separated by 1 standard deviation vs. Gaussian with the same mean and variance

Here the means are separated by $1$ standard deviation and it is harder to separate the mixture from the Gaussian by eye.

Motivation: I disagree with some lazy people about some actual distributions they haven't measured which they assume are close to normal because that would be nice. I'm lazy, too. I don't want to measure the distributions, either. I want to be able to say their assumptions are inconsistent, because they are saying that a finite mixture of Gaussians with different means is a Gaussian which isn't right. I don't just want to say that the asymptotic shape of the tail is wrong because these are just approximations which are only supposed to be reasonably accurate within a few standard deviations of the mean. I'd like to be able to say that if the components are well-approximated by normal distributions then the mixture isn't, and I'd like to be able to quantify this.

I don't know the right distance from normality to use: supremum of differences between CDFs, $L^1$ distance, earth-mover's distance, KL divergence, etc. I'd be happy to get bounds in terms of any of these, or other measures. I would be happy to know the distance to the Gaussian with the same mean and standard deviation as the mixture, or the minimum distance with any Gaussian. If it helps, you may restrict to the case that the mixture is of $2$ Gaussians so that the smaller weight is greater than $1/4$.

Improve this question
$\endgroup$
6
  • 3
    $\begingroup$ If a mixture is very close to nrmal then using a normal approximation is not laziness it is a simplification and could be a good one. But in your example you show a mixture that is flatter than a nromal in the center, more spread in the middle and shorter in the tail when compared to the best approximating normal. I think you would want to look at some sort of integrated difference between the two cdf's. Not the KS measure because the maximu discrepency may not be very large but the average discrepency over a region might be relatively large. $\endgroup$
    – Michael R. Chernick
    Commented Sep 10, 2012 at 15:52
  • 1
    $\begingroup$ Can we assume that there is statistically significant evidence for the mixture of Gaussians over a normal approximation? We only need to worry whether the difference is of practical significance if the difference is known to be statistically significance. Michaels suggestion of something like the Anderson-Darling statistic would be a reasonable place to start. $\endgroup$
    – Dikran Marsupial
    Commented Sep 10, 2012 at 16:18
  • $\begingroup$ @Dikran Marsupial: The population is broken up into subgroups which are known to have different means. The component means are known with high accuracy. The ratio between the differences between the component means and the component standard deviations varies, but may be between $1/2$ and $2$ in some cases of interest, unfortunately not enough to cause the total distribution to be bimodal. $\endgroup$
    – Douglas Zare
    Commented Sep 10, 2012 at 16:34
  • 3
    $\begingroup$ It sounds like you're really asking a model selection question: given some data to model, when should one prefer a Normal distribution compared to a mixture (or more generally, how should one choose the number of mixture components)? Reframing the question like this would give you access to, oh, a few hundred related questions on this site :-). $\endgroup$
    – whuber ♦
    Commented Sep 10, 2012 at 19:21
  • 1
    $\begingroup$ @Xi'an That sounds interesting, but averaged over what? Given the context, where the mixture has been specified, there remains a two-parameter family of normal distributions against which it can be compared. If you adopt some personal prior for that family, then perhaps you do have a measure of dissimilarity. But (1) it becomes a personal distance--essentially a function of the prior--which seems a little bit like begging the question--and (2) would you determine the prior before or after learning what the mixture is? $\endgroup$
    – whuber ♦
    Commented Jan 28, 2015 at 14:34

2 Answers 2

Reset to default
9
$\begingroup$

KL divergence would be natural because you have a natural base distribution, the single Gaussian, from which your mixture diverges. On the the other hand, the KL divergence (or its symmetrical 'distance' form) between two Gaussian mixtures, of which your problem is a special case, seems to be intractable in general. Hershey and Olson (2007) looks like a reasonable summary of the available approximations, including variational methods that may possibly offer easier bounds.

However, if you want to have an argument about the ill-effects of assuming something is Gaussian when it's really a mixture then it's best to have a good idea about the consequences you're actually interested in - something more specific than simply 'being wrong' (this is @Michael-Chernick's point). For example, the consequences for a test, or an interval, or somesuch. Two obvious effects of the mixture are overdispersion, which is pretty much guaranteed, and multimodality, which will confuse maximizers.

Improve this answer
$\endgroup$
1
$\begingroup$

Let me follow up on considering the consequences of incorrect distribution specification. Rather than using a generic measure of distance, such as KL Divergence, you can evaluate a customized measure of "difference", germane to the consequences at hand.

As an example, if the distribution is going to be used for risk calculation, for instance to determine that the probability of failure is low enough, then the only things that matter in the fit are the probability calculations in the extreme tail. This may be relevant to decisions on multi-billion dollar programs, and involve matters of life and death.

Where is the Normal assumption likely to be most inaccurate? In many cases, in the extreme tails, the only place which matters for these crucial risk calculations. If for instance, your true distribution is a mixture of Normals having the same mean, but different standard deviations, then the tails of the mixture distribution are fatter than the tails of the Normal distribution having the same mean and standard deviation. This can easily result in orders of magnitude difference (underestimation of risk) for probabilities in the extreme tail.

So, for instance, at a crucial level $U$, the relevant measure of difference might be $P(X_{Mixture} > U) - P(X_{Normal} > U)$. In such case, it doesn't matter how good the agreement is in the rest of the distribution.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.