4
$\begingroup$

Suppose that we have a mixture model:

$$ p_\theta(y) = \sum_{k = 1}^{K}w_k \phi(y;\mu_k, \sigma^2_k) $$

where $\phi(y;\mu_k, \sigma^2_k)$ is the normal density at $y$ with mean $\mu$ and variance $\sigma^2$. $\theta$ contains the weights, means, and variances. Why is it the case here that the model is not identifiable?

I know by definition not being identifiable just means that:

$$ \forall \theta_1 \neq \theta_2 \ \ \ \ , \ \ \ p_{\theta_1} =p_{\theta_2} $$

What is an explicit example here where this holds or why is it bad if it is unidentifiable? Thanks.

$\endgroup$
1
6
$\begingroup$

Consider the case where $\theta_1 = (w_1=0.5, \mu_1 = 0, \sigma_1^2 = 1)$ and $\theta_2 = (w_2=0.5, \mu_2 = 1, \sigma_2^2 = 1)$ We get exactly the same fit to the data if $\theta_1 = (w_1=0.5, \mu_1 = 1, \sigma_1^2 = 1)$ and $\theta_2 = (w_2=0.5, \mu_2 = 0, \sigma_2^2 = 1)$ Thus, there is no way to empirically learn the value of $\mu_1$ regardless of the amount of data (i.e., it is not identified).

In this case, the absence of identifiability is not "bad", as the real problem to be solved is estimating the parameters, and whether we choose one component of the mixture the first or second component is immaterial.

$\endgroup$
4
$\begingroup$

The mixture sum is well-defined and almost always identifiable, while the elements of the sum can be switched with one another without changing the sum, by mere commutativity: a+b=b+a.

For instance, here is the surface of a log-likelihood associated with the mixture$$\frac{1}{2}\mathrm{N}(\mu_1,1)+\frac{1}{2}\mathrm{N}(\mu_2,1)$$ and 250 observations from this distribution where I used $\mu_1=0$ and $\mu_2=2$. (Or equivalently $\mu_2=0$ and $\mu_1=2$.) The paths on the surface are EM steps with different starting points, but the figure shows clearly the symmetry of the likelihood function along the diagonal, which is an illustration of this lack of identifiability of $\mu_1$ as such.

enter image description here

[The picture is taken from our book Introduction Monte Carlo Method with R, written with my late friend George Casella.]

The lack of identifiability is not an issue at the level of the distribution but it creates inference problems, from a multimodal likelihood to complex limiting distributions for the estimators, to exploration troubles for numerical solutions.

Here is for instance the output of four STAN chains ending up in different modes of the posterior, taken from a recent discussion on the identifiability of mixtures by Michael Betancourt:

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.