# Meaning of Gaussian mixture model parameters

I came across this question from a tutorial:

Suppose we have observations $x_1$ , $x_2$ , $\ldots$, $x_n$ of a continuous r.v. $X$ known to be drawn from a “mixture” of $k$ Gaussian distributions. Assuming the set of parameters to be $\theta$, give an expression for $p(x_i | \theta)$ (this is the likelihood of $x_i$ given $\theta$)

My question here is: What is $\theta$? Each Guassian in the mixture will have a mean and variance, and a prior probability. What does $\theta$ mean then?

The answer to the question is: $\sum_{j=1}^k P(x_i | j,\theta)P(j | \theta)$. Can someone explain to me how it is calculated?

In this context, $\theta$ denotes the set of all parameters of the model. These are the means of the individual gaussians ($\mu_j$), their variances ($\sigma_j$), and also the weights of each gaussian ($\pi_j$), i.e. how important is that particular component in the whole mixture. Therefore,

$$\theta = (\mu_1, \sigma_1, \pi_1 \ldots, \mu_k, \sigma_k, \pi_k).$$

When evaluating the probability of a sample $x_i$ under the distribution modeled by the mixture, $P(x_i|\theta)$, you first need to marginalize out the cluster assignment (which is unknown), that is why you sum over all $j$s:

$$P(x_i|\theta) = \sum_{j=1}^k P(x_i|\theta, j) P(j|\theta)$$

Now, $P(j|\theta) = \pi_j$ and $P(x_i|\theta, j)=P(x_i|\mu_j, \sigma_j)$.

• If that is so, then what is the difference b/w P(xi | j,$\theta$) and P(xi | j)? Adding theta to the evidence shouldn't make any difference, since those parameters would be use automatically when we use the jth Gaussian as evidence. Am I correct? – Prashant Pandey May 8 '18 at 8:40
• $P(x_i|j)$ does not make sense. $P(x_i|\theta,j)$ is equal to $P(x_i|\mu_j, \sigma_j)$. – Jan Kukacka May 8 '18 at 8:43
• Oh, makes perfect sense now. – Prashant Pandey May 8 '18 at 8:48
• I added some extra explanation to the answer. – Jan Kukacka May 8 '18 at 8:50

Gaussian mixture means, you have a set of $k$ Gaussian distributions that have a parameter $\theta$. We denote them by $P(x|\theta, j)$, where $j \in \{1,...,k\}$ is the index of the Gaussian. Each of these Gaussian distributions - actually each of the indices - has some probability. The probability of $j \in \{1,....,k\}$ by $P(j|\theta)$. The model you give here means: First sample $j^* ~ P(\cdot|\theta)$, then sample $x \sim P( \cdot | \theta, j^*)$.

The parameter $\theta$ could include means and covariances of the Gaussians, as well as the probabilities of each Gaussian.

• Oh alright. So, each Gaussian has its own $\theta$, right? – Prashant Pandey May 8 '18 at 7:56
• Probably, that depends on the model. You could have $\theta =(p_1, m_1, \sigma^2_1, p_2, m_2, \sigma^2_2,..., p_k,m_k,\sigma^2_k),$ where $P(x|\theta, j) = N(x; m_j, \sigma^2_j)$ and $P(j|\theta) = p_j$, for $j =1,...,k.$ – Jonas May 8 '18 at 9:27