1
$\begingroup$

I came across this question from a tutorial:

Suppose we have observations $x_1$ , $x_2$ , $\ldots$, $x_n$ of a continuous r.v. $X$ known to be drawn from a “mixture” of $k$ Gaussian distributions. Assuming the set of parameters to be $\theta$, give an expression for $p(x_i | \theta)$ (this is the likelihood of $x_i$ given $\theta$)

My question here is: What is $\theta$? Each Guassian in the mixture will have a mean and variance, and a prior probability. What does $\theta$ mean then?

The answer to the question is: $\sum_{j=1}^k P(x_i | j,\theta)P(j | \theta)$. Can someone explain to me how it is calculated?

$\endgroup$
1
$\begingroup$

In this context, $\theta$ denotes the set of all parameters of the model. These are the means of the individual gaussians ($\mu_j$), their variances ($\sigma_j$), and also the weights of each gaussian ($\pi_j$), i.e. how important is that particular component in the whole mixture. Therefore,

$$\theta = (\mu_1, \sigma_1, \pi_1 \ldots, \mu_k, \sigma_k, \pi_k).$$

When evaluating the probability of a sample $x_i$ under the distribution modeled by the mixture, $P(x_i|\theta)$, you first need to marginalize out the cluster assignment (which is unknown), that is why you sum over all $j$s:

$$P(x_i|\theta) = \sum_{j=1}^k P(x_i|\theta, j) P(j|\theta)$$

Now, $P(j|\theta) = \pi_j$ and $P(x_i|\theta, j)=P(x_i|\mu_j, \sigma_j)$.

$\endgroup$
  • $\begingroup$ If that is so, then what is the difference b/w P(xi | j,$\theta$) and P(xi | j)? Adding theta to the evidence shouldn't make any difference, since those parameters would be use automatically when we use the jth Gaussian as evidence. Am I correct? $\endgroup$ – Prashant Pandey May 8 '18 at 8:40
  • $\begingroup$ $P(x_i|j)$ does not make sense. $P(x_i|\theta,j)$ is equal to $P(x_i|\mu_j, \sigma_j)$. $\endgroup$ – Jan Kukacka May 8 '18 at 8:43
  • $\begingroup$ Oh, makes perfect sense now. $\endgroup$ – Prashant Pandey May 8 '18 at 8:48
  • $\begingroup$ I added some extra explanation to the answer. $\endgroup$ – Jan Kukacka May 8 '18 at 8:50
1
$\begingroup$

Gaussian mixture means, you have a set of $k$ Gaussian distributions that have a parameter $\theta$. We denote them by $P(x|\theta, j)$, where $j \in \{1,...,k\}$ is the index of the Gaussian. Each of these Gaussian distributions - actually each of the indices - has some probability. The probability of $j \in \{1,....,k\}$ by $P(j|\theta)$. The model you give here means: First sample $j^* ~ P(\cdot|\theta)$, then sample $x \sim P( \cdot | \theta, j^*)$.

The parameter $\theta$ could include means and covariances of the Gaussians, as well as the probabilities of each Gaussian.

$\endgroup$
  • $\begingroup$ Oh alright. So, each Gaussian has its own $\theta$, right? $\endgroup$ – Prashant Pandey May 8 '18 at 7:56
  • $\begingroup$ Probably, that depends on the model. You could have $\theta =(p_1, m_1, \sigma^2_1, p_2, m_2, \sigma^2_2,..., p_k,m_k,\sigma^2_k),$ where $P(x|\theta, j) = N(x; m_j, \sigma^2_j)$ and $P(j|\theta) = p_j$, for $j =1,...,k.$ $\endgroup$ – Jonas May 8 '18 at 9:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.