Gaussian clusters and original distributions

Question

In Gaussian clustering (i.e. General Mixture Models) we model the data with some clusters. For example, in the below figure, we have two clusters $C_1, C_2$, each of which are modeled with a Gaussian (Normal) distribution: $p_{C_1} \sim \mathcal{N}(\mu_1, \Sigma_1)$ and $p_{C_2} \sim \mathcal{N}(\mu_2, \Sigma_2)$ with proportions $\pi_1$ and $\pi_2$.

The probability that a given point $x$ belongs to $C_1$ (is generated from $C_1$) is: $$ p(C_1 | x ) = \frac { \pi_1 \mathcal{N}(\mu_1, \Sigma_1)} {\sum_{i=1}^2{ \pi_i \mathcal{N}(\mu_i, \Sigma_i) } } $$ The probability that $x$ belongs to $C_2$ can be similarly calculated.

Now, assume that we have a group of points that are generated (sampled) from another Gaussian distribution $q(x) \sim \mathcal{N}(\mu_q, \Sigma_q)$.

Obviously, we can compute the probability of each point belonging to each cluster. However, we instead want to compute the expected probability of a sample belonging to each cluster. In other words, if we generate infinite samples from the $q(x)$, what is the average (expected) value of $p(C_1 | x )$ for all points?

**P.S. ** I need the solution in closed-form.

Answer 1

I think you are assuming that the parameters $(\pi_j, \mu_j, \Sigma_j: q=1,2)$ are known and that the sample $\mathbf{X}=(X_1, \ldots, X_n)$ is independently identically distributed (i.i.d.) from a Gaussian distribution $\mathcal{N}(\mu_3, \Sigma_3)$ according to the true data generating distribution.

However, your model assumes that $$ X_i \overset{iid}{\sim} \sum_{j=1}^2 \pi_j \mathcal{N}(\mu_j, \Sigma_j)$$

Thus, the probability that a given point $x_i$ belongs to the first Gaussian component (according to your model), i.e. is assigned in cluster $C_1$ is: $$ p(x_i \in C_1) = \frac { \pi_1 \mathcal{N}(x_i \mid \mu_1, \Sigma_1)} {\sum_{j=1}^2{ \pi_j \mathcal{N}(x_i \mid \mu_j, \Sigma_j) } }, $$ where $\mathcal{N}(x \mid \mu, \Sigma)$ is the Gaussian density with mean vector $\mu$ and variance matrix $\Sigma$ evaluated in the vector $x \in \mathbb{R}^2$.

Moreover, since all the parameters of the model are fixed and known the mean (with respect to the data generating truth) probability that the random variable point $X_i$ belongs to the first Gaussian component (according to your model), i.e. is assigned in cluster $C_1$ is $$ p_1 := \mathrm{E}[p(X_i \in C_1)] = \int p(x \in C_1) \mathcal{N}(x \mid \mu_3, \Sigma_3) \mathrm{d}x, $$ that can be computed by substituting the explicit expression of the Gaussian densities involved.

Then, thanks to the independence assumption (in the data generating truth), the mean (with respect to the data generating truth) probability that all the observations $\mathbf{X}$ are assigned to the first component (in cluster $C_1$) is $p:=p_1^n$.

Note that, since $p_1\in (0,1)$ such a probability $p$ goes to $0$ as the sample size $n$ goes to infinity (even if the data generating truth is equal to the first Gaussian component!).

Answer 2

Well it's clearly

$$ \int p(x|C_i) q(x)\, dx $$ but wouldn't you rather have $P(C_i | x)$ and the expected values of these, i.e. probabilities of classification?