Gaussian mixture model - does an improper uniform prior give a proper posterior?

Question

We draw $n$ i.i.d. points $x_1 , x_2 , ..., x_n$ from the following Gaussian mixture:

$$p(x|\mu_1,\mu_2) = \frac{1}{2} \text{N} (x|\mu_1,1) + \frac{1}{2} \text{N} (x|\mu_2,1).$$

The prior is the improper prior $p(\mu_1, \mu_2) \propto 1$. Does the prior $\times$ likelihood product have a finite integral (i.e., is the posterior defined)?

Answer 1

Under an improper product prior $\pi(\mu_1,\mu_2)=\pi_1(\mu_1)\pi_2(\mu_2)$ and no further restriction on the parameter $(\mu_1,\mu_2)$ the posterior is improper:

Indeed, the likelihood $$L_\mathbf{x}(\mu_1,\mu_2) = (8 \pi)^{-n/2} \prod_{i=1}^n \Bigg[ \exp \Big( -\frac{1}{2} (x_i-\mu_1)^2 \Big) + \exp \Big( -\frac{1}{2} (x_i-\mu_2)^2 \Big) \Bigg].$$ can be decomposed as the sum over all partitions of the sample $(x_1,\ldots,x_n)$ into two groups: $$L_\mathbf{x}(\mu_1,\mu_2) = (8 \pi)^{-n/2}\sum_{\sigma=1}^{2^n} \exp \Big( -\frac{1}{2} \sum_{i\in p_1(\sigma)}(x_i-\mu_1)^2 \Big) \exp \Big( -\frac{1}{2} \sum_{i\in p_2(\sigma)}(x_i-\mu_2)^2 \Big) .$$ This decomposition includes the partition $p(\sigma_0)$ where all elements of the sample are allocated to the first group $p_1(\sigma_0)=\{1,\ldots,n\}$ and $p_2(\sigma_0)=\emptyset$, therefore $$L_\mathbf{x}(\mu_1,\mu_2) \ge (8 \pi)^{-n/2} \exp \Big( -\frac{1}{2} \sum_{i=1}^n(x_i-\mu_1)^2 \Big)$$ and this term does not involve $\mu_2$. This implies that against any (improper prior) infinite-mass measure $\pi_2(\text{d}\mu_2)$ on $\mu_2$, we have $$\int L_\mathbf{x}(\mu_1,\mu_2) \pi_2(\text{d}\mu_2) \ge \int (8 \pi)^{-n/2} \exp \Big( -\frac{1}{2} \sum_{i=1}^n(x_i-\mu_1)^2 \Big) \pi_2(\text{d}\mu_2) = +\infty$$hence that the posterior is not defined.

We recently published two papers on this very topic:

1. one with Clara Grazian where we show that the Jeffreys prior and posterior distributions are mostly improper.
2. one with Kaniav Kamary and Kate Lee where we propose a family of improper priors that lead to proper posteriors.

Side story: In my first work on mixtures with Jean Diebolt (written in 1990, published in 1994!), we restricted the partition space to the sole partitions that allowed for integrable terms. Which permitted the use of improper priors, but changed the distribution of the sample by introducing a slight correlation between the elements of the sample.

Answer 2

From your specified model you have likelihood function:

$$L_\mathbf{x}(\mu_1,\mu_2) = (8 \pi)^{-n/2} \prod_{i=1}^n \Bigg[ \exp \Big( -\frac{1}{2} (x_i-\mu_1)^2 \Big) + \exp \Big( -\frac{1}{2} (x_i-\mu_2)^2 \Big) \Bigg].$$

To facilitate the analysis, let $\mathscr{B} \equiv \{ 1,2 \}^n$ be the set of all $n$-length binary vectors of values of the two means. For any vector $\mathbf{b} \in \mathscr{B}$ let $\mathcal{S}(\mathbf{b}) \equiv \{ i=1,...,n | b_i=1 \}$ be the set of index values that use the first mean. With the improper uniform prior $\pi (\mu_1, \mu_2) \propto 1$ you get the posterior:

$$\begin{equation} \begin{aligned} \pi (\mu_1, \mu_2 | \mathbf{x}) &\propto L_\mathbf{x}(\mu_1,\mu_2) \\[6pt] &\propto \prod_{i=1}^n \Bigg[ \exp \Big( -\frac{1}{2} (x_i-\mu_1)^2 \Big) + \exp \Big( -\frac{1}{2} (x_i-\mu_2)^2 \Big) \Bigg] \\[6pt] &= \sum_{\mathbf{b} \in \mathscr{B}} \exp \Big( -\frac{1}{2} \sum_{i=1}^n (x_i-\mu_{b_i})^2 \Big) \\[6pt] &= \sum_{\mathbf{b} \in \mathscr{B}} \exp \Big( -\frac{1}{2} \sum_{i \in \mathcal{S}(\mathbf{b})} (x_i-\mu_1)^2 \Big) \exp \Big( -\frac{1}{2} \sum_{i \notin \mathcal{S}(\mathbf{b})} (x_i-\mu_2)^2 \Big). \\[6pt] \end{aligned} \end{equation}$$

For the vectors $\mathbf{b} = (1,...,1)$ and $\mathbf{b} = (2,...,2)$ we get empty classes over one of the inner sums and so the outer summation includes one term that does not depend on $\mu_1$ and one term that does not depend on $\mu_2$. For all other terms (i.e., with some values from each mean) we can simplify as follows. Letting $\bar{x}_1^{(k)} (\mathbf{b}) \equiv \sum_{i \in \mathcal{S}(\mathbf{b})} x_i^k$ and $\bar{x}_2^{(k)} (\mathbf{b}) \equiv \sum_{i \notin \mathcal{S}(\mathbf{b})} x_i^k$ we have:

$$\begin{equation} \begin{aligned} \exp \Big( -\frac{1}{2} \sum_{i \in \mathcal{S}(\mathbf{b})} (x_i-\mu_1)^2 \Big) &= \exp \Big( -\frac{1}{2} \Big( \mu_1^2 + 2 \mu_1 \sum_{i \in \mathcal{S}(\mathbf{b})} x_i + \sum_{i \in \mathcal{S}(\mathbf{b})} x_i^2 \Big) \Big) \\[6pt] &= \exp \Big( -\frac{1}{2} ( \mu_1^2 + 2 \mu_1 \bar{x}_1^{(1)} (\mathbf{b}) + \bar{x}_1^{(2)} (\mathbf{b}) ) \Big) \\[10pt] &= \exp \Big( -\frac{1}{2} ( \bar{x}_1^{(2)} (\mathbf{b}) - \bar{x}_1^{(1)} (\mathbf{b})^2 ) \Big) \exp \Big( -\frac{1}{2} (\mu_1 - \bar{x}_1^{(1)} (\mathbf{b}))^2 \Big) \\[10pt] &\propto \exp \Big( -\frac{1}{2} ( \bar{x}_1^{(2)} (\mathbf{b}) - \bar{x}_1^{(1)} (\mathbf{b})^2 ) \Big) \cdot \text{N}(\mu_1 | \bar{x}_1^{(1)} (\mathbf{b}), 1), \\[10pt] \exp \Big( -\frac{1}{2} \sum_{i \notin \mathcal{S}(\mathbf{b})} (x_i-\mu_2)^2 \Big) &\propto \exp \Big( -\frac{1}{2} ( \bar{x}_2^{(2)} (\mathbf{b}) - \bar{x}_2^{(1)} (\mathbf{b})^2 ) \Big) \cdot \text{N}(\mu_2 | \bar{x}_2^{(1)} (\mathbf{b}), 1). \\[10pt] \end{aligned} \end{equation}$$

Let $\bar{\mathscr{B}}$ denote the vectors that have mixtures of values from both parts of the mixture (i.e., excluding $\mathbf{b} = (1,...,1), (2,...,2)$). Let $H(\mathbf{b}) \equiv \bar{x}_1^{(2)} (\mathbf{b}) - \bar{x}_1^{(1)} (\mathbf{b})^2 + \bar{x}_2^{(2)} (\mathbf{b}) - \bar{x}_2^{(1)} (\mathbf{b})^2$ and we then have:

$$\begin{equation} \begin{aligned} \pi (\mu_1, \mu_2 | \mathbf{x}) &\propto \exp \Big( -\frac{1}{2} \sum_{i=1}^n (x_i-\mu_1) \Big) + \exp \Big( -\frac{1}{2} \sum_{i=1}^n (x_i-\mu_2) \Big) \\[6pt] &+ \sum_{\mathbf{b} \in \bar{\mathscr{B}}} \exp \Big( -\frac{1}{2} H(\mathbf{b}) \Big) \cdot \text{N}(\mu_1 | \bar{x}_1^{(1)} (\mathbf{b}), 1) \cdot \text{N}(\mu_2 | \bar{x}_2^{(1)} (\mathbf{b}), 1). \\[6pt] \end{aligned} \end{equation}$$

This kernel has an infinite integral, owing to integration over the terms that do not depend on both mean parameters. Hence, the posterior is improper. (Thanks to Xi'an for pointing out an error in the previous version of this answer.)