Have some confusion on Gaussian mixture distribution model (reference is from the book of Pattern Recognition and Machine Learning). My confusion is how below formula works?


My thought is, $z_1$ could be $0$ or $1$, $z_2$ could be $0$ or $1$, ... $z_K$ could be $0$ or $1$, how could sum of them (them I mean $z_1$, $z_2$, ..., $z_K$) to be 1?

enter image description here

  • 3
    $\begingroup$ $z_k$ is a binary component of a $K$ dimensional vector, $z$. The purpose of $z$ is to indicate which one of the $K$ populations a particular observation comes from. Under the model, an observational element must belong to one and only one of the $K$ populations. Hence the summation to 1. $\endgroup$
    – lmo
    Nov 25, 2016 at 20:39
  • $\begingroup$ Imo is correct but to complete the argument the probability that zk is 1 is the proportion that corresponds to that specific normal distribution in the mixture. $\endgroup$ Nov 25, 2016 at 23:29
  • 2
    $\begingroup$ "binary component" means that it is binary, either 0 or 1, and that it is one element in the vector. It is more or less a combination of two math terms, but probably with a computer science influence. To emphasize, let's say $z = (0, 1, 0, 0, 0)$. Then $z_1=0$, $z_2=1$, and $z_k=0$ for $k \in \{1, 3, 4, 5 \}$. $\endgroup$
    – lmo
    Nov 26, 2016 at 13:01
  • 1
    $\begingroup$ That is correct. $\endgroup$
    – lmo
    Nov 27, 2016 at 13:42
  • 1
    $\begingroup$ just fyi it's usually called the one-hot representation $\endgroup$
    – dontloo
    Dec 1, 2016 at 9:18

1 Answer 1


For each $x_i$ in the sample $(x_1,\ldots,x_n)$ one can create an binary (and unobserved) vector $\mathbf{z}_i=(z_{i1},\ldots,z_{iK})$ made of components in $\{0,1\}$ such that one and only one of the $z_{ij}$'s is equal to one, e.g., $\mathbf{z}_i=(0,\ldots,0,1,0,\ldots,0)$, and all others are equal to zero. The corresponding $j$ is called the component of $x_i$. Conditional on $\mathbf{z}_i$, $x_i$ is distributed as a normal variate $$\mathcal{N}\left(\sum_{j=1}^K z_{ij}\mu_j,\sum_{j=1}^K z_{ij}\Sigma_j\right)$$Marginally, the vector $\mathbf{z}_i$ is distributed as a Multinomial $\mathcal{M}_K(1;\pi_1,\ldots,\pi_k)$ variate.

  • $\begingroup$ Thanks Xi'an, vote up, what means $\mu_j$ and $\sum_j$ in your formula? $\endgroup$
    – Lin Ma
    Dec 1, 2016 at 7:32
  • 1
    $\begingroup$ This is the same notation as in formula (9.7). $\endgroup$
    – Xi'an
    Dec 1, 2016 at 8:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.