2
$\begingroup$

Here's a scenario: Suppose I collect a dataset $\{x_i\}_{i=1}^k\subseteq\mathbb R$ of data points $x_i$, and I wish to explain it using a mixture of two Gaussians; assume the unknown parameters are the means $\mu_1$ and $\mu_2$ of the mixture model. Following Bayesian methodology, I formulate some posterior distribution $P((\mu_1,\mu_2)|\{x_i\}_i)$, from which I can use Monte Carlo methods to sample lots of likely $(\mu_1^j,\mu_2^j)$ pairs.

The problem is, swapping $(\mu_1,\mu_2)$ to $(\mu_2,\mu_1)$ describes the same mixture of Gaussians; a symmetry group---in this case $\mathbb{Z}/2$---acts on the random samples I'm generating without changing their interpretation. This prevents me, for example, from directly averaging the pairs that come out of my sampler. That is, the expectation $\mathbb E_{(\mu_1,\mu_2)\sim P(\cdots)}[(\mu_1,\mu_2)]$ really doesn't make sense since the ordering of the two elements is arbitrary in each sample.

Is there a general mathematical/statistical framework for coping with group-invariant probability measures? What inference algorithms do people use in this scenario? Do notions like mean/covariance extend to quotient spaces?

Any pointers are much appreciated. This seems like a fundamental/classical problem, and likely I'm just missing Google search terms! My apologies if I have abused "Bayesian language" here---this is not my typical research field.

$\endgroup$
  • 1
    $\begingroup$ Usually one just chooses a fundamental domain for the group action. $\endgroup$ – whuber Feb 28 at 0:08
  • $\begingroup$ Afraid I don’t know what that means! Might you have a pointer? Do statistical theorems carryover to this case? $\endgroup$ – Justin Solomon Feb 28 at 0:35
  • $\begingroup$ en.wikipedia.org/wiki/Fundamental_domain $\endgroup$ – whuber Feb 28 at 15:08
  • $\begingroup$ Sure, this amounts to reasoning about the quotient space directly. But notions like expectation are weird in this quotient. For example, using [0,2pi) to represent the unit circle neglects the fact that 2pi=0 (mod 2pi), which gets in the way of computing expectations and other quantities. $\endgroup$ – Justin Solomon Feb 28 at 17:39
  • $\begingroup$ Since you are using Bayesian methodology, I don't see why you even need to compute expectations: aren't you interested in the posterior distribution? Use properties of that distribution that are suitable for your analysis, such as modes and measures of spread around those modes. $\endgroup$ – whuber Feb 28 at 18:37
2
$\begingroup$

$\qquad\qquad\qquad$enter image description here

This is a well-documented issue with mixture modelling, although in your case of having only the means unknown, they are identifiable when the weights in the mixture differ from $(1/2,1/2)$. A first branch of results in this area follow the 1963 paper of Teicher. And indeed a plain Bayesian approach under exchangeable prior assumptions cannot handle using a posterior mean since both marginals are identical. Another branch of results attempts at restricting the parameter space to make it identifiable. From strong truncation imposing ordering to representing the posterior as a point process as described in the first chapter by Peter Green in our very recent Handbook of Mixture Analysis, to isolate a single mode in the posterior distribution by fighting label switching, as also discussed in this X validated question, and advocated by Geweke (2007). See also my entries on this topic on my blog.

$\endgroup$
  • 1
    $\begingroup$ Awesome! This definitely seemed like a topic that must have been studied in the past. I really appreciate these pointers and will take a close look! $\endgroup$ – Justin Solomon Feb 28 at 13:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.