What is the distance between a finite Gaussian mixture and a Gaussian?

Suppose I have a mixture of finitely many Gaussians with known weights, means, and standard deviations. The means are not equal. The mean and standard deviation of the mixture can be calculated, of course, since the moments are weighted averages of the moments of the components. The mixture is not a normal distribution, but how far from normal is it?

The above image shows the probability densities for a Gaussian mixture with component means separated by $2$ standard deviations (of the components) and a single Gaussian with the same mean and variance.

Here the means are separated by $1$ standard deviation and it is harder to separate the mixture from the Gaussian by eye.

Motivation: I disagree with some lazy people about some actual distributions they haven't measured which they assume are close to normal because that would be nice. I'm lazy, too. I don't want to measure the distributions, either. I want to be able to say their assumptions are inconsistent, because they are saying that a finite mixture of Gaussians with different means is a Gaussian which isn't right. I don't just want to say that the asymptotic shape of the tail is wrong because these are just approximations which are only supposed to be reasonably accurate within a few standard deviations of the mean. I'd like to be able to say that if the components are well-approximated by normal distributions then the mixture isn't, and I'd like to be able to quantify this.

I don't know the right distance from normality to use: supremum of differences between CDFs, $L^1$ distance, earth-mover's distance, KL divergence, etc. I'd be happy to get bounds in terms of any of these, or other measures. I would be happy to know the distance to the Gaussian with the same mean and standard deviation as the mixture, or the minimum distance with any Gaussian. If it helps, you may restrict to the case that the mixture is of $2$ Gaussians so that the smaller weight is greater than $1/4$.

• If a mixture is very close to nrmal then using a normal approximation is not laziness it is a simplification and could be a good one. But in your example you show a mixture that is flatter than a nromal in the center, more spread in the middle and shorter in the tail when compared to the best approximating normal. I think you would want to look at some sort of integrated difference between the two cdf's. Not the KS measure because the maximu discrepency may not be very large but the average discrepency over a region might be relatively large. – Michael R. Chernick Sep 10 '12 at 15:52
• Can we assume that there is statistically significant evidence for the mixture of Gaussians over a normal approximation? We only need to worry whether the difference is of practical significance if the difference is known to be statistically significance. Michaels suggestion of something like the Anderson-Darling statistic would be a reasonable place to start. – Dikran Marsupial Sep 10 '12 at 16:18
• @Dikran Marsupial: The population is broken up into subgroups which are known to have different means. The component means are known with high accuracy. The ratio between the differences between the component means and the component standard deviations varies, but may be between $1/2$ and $2$ in some cases of interest, unfortunately not enough to cause the total distribution to be bimodal. – Douglas Zare Sep 10 '12 at 16:34
• It sounds like you're really asking a model selection question: given some data to model, when should one prefer a Normal distribution compared to a mixture (or more generally, how should one choose the number of mixture components)? Reframing the question like this would give you access to, oh, a few hundred related questions on this site :-). – whuber Sep 10 '12 at 19:21
• @whuber: the distance to the normal could then be expressed as the (average) power of a test aiming at separating mixture from single Gaussian. – Xi'an Jan 28 '15 at 13:32

So, for instance, at a crucial level $U$, the relevant measure of difference might be $P(X_{Mixture} > U) - P(X_{Normal} > U)$. In such case, it doesn't matter how good the agreement is in the rest of the distribution.