3
$\begingroup$

In Gaussian clustering (i.e. General Mixture Models) we model the data with some clusters. For example, in the below figure, we have two clusters $C_1, C_2$, each of which are modeled with a Gaussian (Normal) distribution: $p_{C_1} \sim \mathcal{N}(\mu_1, \Sigma_1)$ and $p_{C_2} \sim \mathcal{N}(\mu_2, \Sigma_2)$ with proportions $\pi_1$ and $\pi_2$.

enter image description here

The probability that a given point $x$ belongs to $C_1$ (is generated from $C_1$) is: $$ p(C_1 | x ) = \frac { \pi_1 \mathcal{N}(\mu_1, \Sigma_1)} {\sum_{i=1}^2{ \pi_i \mathcal{N}(\mu_i, \Sigma_i) } } $$ The probability that $x$ belongs to $C_2$ can be similarly calculated.

Now, assume that we have a group of points that are generated (sampled) from another Gaussian distribution $q(x) \sim \mathcal{N}(\mu_q, \Sigma_q)$.

enter image description here

Obviously, we can compute the probability of each point belonging to each cluster. However, we instead want to compute the expected probability of a sample belonging to each cluster. In other words, if we generate infinite samples from the $q(x)$, what is the average (expected) value of $p(C_1 | x )$ for all points?

**P.S. ** I need the solution in closed-form.

$\endgroup$
0
1
$\begingroup$

I think you are assuming that the parameters $(\pi_j, \mu_j, \Sigma_j: q=1,2)$ are known and that the sample $\mathbf{X}=(X_1, \ldots, X_n)$ is independently identically distributed (i.i.d.) from a Gaussian distribution $\mathcal{N}(\mu_3, \Sigma_3)$ according to the true data generating distribution.

However, your model assumes that $$ X_i \overset{iid}{\sim} \sum_{j=1}^2 \pi_j \mathcal{N}(\mu_j, \Sigma_j)$$

Thus, the probability that a given point $x_i$ belongs to the first Gaussian component (acording to your model), i.e. is assigned in cluster $C_1$ is: $$ p(x_i \in C_1) = \frac { \pi_1 \mathcal{N}(x_i \mid \mu_1, \Sigma_1)} {\sum_{j=1}^2{ \pi_j \mathcal{N}(x_i \mid \mu_j, \Sigma_j) } }, $$ where $\mathcal{N}(x \mid \mu, \Sigma)$ is the Gaussian density with mean vector $\mu$ and variance matrix $\Sigma$ evaluated in the vector $x \in \mathbb{R}^2$.

Moreover, since all the parameters of the model are fixed and known the mean (with respect to the data generating truth) probability of that the random variable point $X_i$ belongs to the first Gaussian component (according to your model), i.e. is assigned in cluster $C_1$ is $$ p_1 := \mathrm{E}[p(X_i \in C_1)] = \int p(x \in C_1) \mathcal{N}(x \mid \mu_3, \Sigma_3) \mathrm{d}x, $$ that can be computed substituting the explicit expression of the Gaussian densities involved.

Then, thanks to the independence assumption (in the data generating truth), the mean (with respect to the data generating truth) probability that all the observations $\mathbf{X}$ are assigned to the first component (in cluster $C_1$) is $p:=p_1^n$.

Note that, since $p_1\in (0,1)$ such a probability $p$ goes to $0$ as the sample size $n$ goes to infinity (even if the data generating truth is equal to the first Gaussian component!).

$\endgroup$
0
$\begingroup$

Well it's clearly

$$ \int p(x|C_i) q(x)\, dx $$ but wouldn't you rather have $P(C_i | x)$ and the expected values of these, i.e. probabilities of classification?

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.