Finding number of gaussians in a finite mixture with Wilks' theorem?

Question

Assume I have a set of independent, identically distributed univariate observations $x$ and two hypotheses about how $x$ was generated:

$H_0$: $x$ is drawn from a single Gaussian distribution with unknown mean and variance.

$H_A$: $x$ is drawn from a mixture of two Gaussians with unknown mean, variance and mixing coefficient.

If I understand correctly, these are nested models since the model that $H_0$ represents can be described in terms of $H_A$ if you constrain the parameters of the two Gaussians to be identical or constrain the mixing coefficient to be zero for one of the two Gaussians.

Therefore, it seems like you should be able to use the E-M algorithm to estimate the parameters of $H_A$ and then use Wilks' Theorem to determine whether the likelihood of the data under $H_A$ is significantly greater than that under $H_0$. There's a small leap of faith in the assumption that the E-M algorithm will converge to the maximum likelihood here, but it's one I'm willing to make.

I tried this in a monte carlo simulation, assuming that $H_A$ has 3 more degrees of freedom than $H_0$ (the mean and variance for the second Gaussian and the mixing parameter). When I simulated data from $H_0$, I got a P-value distribution that was substantially non-uniform and enriched for small P-values. (If E-M weren't converging to the true maximum likelihood, the exact opposite would be expected.) What's wrong with my application of Wilks' theorem that's creating this bias?

Answer 1

With a careful specification of how the null hypothesis is contained in the two-component mixture model, it is possible to see what the problem could be. If the five parameters in the mixture model are $\mu_1, \mu_2, \sigma_1, \sigma_2, \rho$, then $$H_0: (\mu_1 = \mu_2 \text{ and } \sigma_1 = \sigma_2) \text{ or } \rho \in \{0, 1\}.$$ because either the two normal mixture components are equal, in which case the mixture proportion $\rho$ is irrelevant, or the mixture proportion $\rho$ is 0 or 1, in which case one of the mixture components is irrelevant. The conclusion is that the null hypothesis can not be specified, not even locally, as a simple parameter restriction that drops the dimension of the parameter space from 5 to 2.

The null hypothesis is a complicated subset of the full parameter space, and under the null the parameters are not even identifiable. The usual assumptions needed to obtain Wilk's theorem break down, most notably it is not possible to construct a proper Taylor expansion of the log-likelihood.

I don't have any personal experience with this particular problem, but I know of other cases where parameters "disappear" under the null, which seems to be the case here as well, and in these cases the conclusions of Wilk's theorem break down too. A quick search gave, among other things, this paper that looks relevant, and where you might be able to find further references on the use of the likelihood ratio test in relation to mixture models.

Answer 2

Inference on the number of mixing components does not satisfy the needed regularity conditions for Wilks theorem since (a) the parameter $\rho$ is on the boundary of the parameter space and (b) the parametrisation is unidentifiable under the null. This is not to say that the distribution of the generalized likelihood ratio is unknown! If all the 5 parameters in your setup are unknown, and more importantly- unbounded- then the distribution of the LR statistic does not converge. If all the unidentifiable parameters are bounded, then the LR statistic is monotone in the supremum of a truncated Gaussian process. The covariance of which is not easy to compute in the general (5 parameter) case, and even when you have it- the distribution of the supremum of such a process is not easily approximated. For some practical results regarding the two-component mixture see here. Interestingly, the paper shows that in rather simple setups, the LR statistic is actually less powerful than some simpler statistics. For the seminal paper on deriving the asymptotic distribution in such problems see here. For all practical purposes, you can fit the mixture using an EM, and then Bootstrap the distribution of the LR statistic. This might take some time as the EM is known to be slow, and you need many replication to capture the effect of the sample size. See here for details.