Have some confusion on Gaussian mixture distribution model (reference is from the book of Pattern Recognition and Machine Learning). My confusion is how below formula works?


My thought is, $z_1$ could be $0$ or $1$, $z_2$ could be $0$ or $1$, ... $z_K$ could be $0$ or $1$, how could sum of them (them I mean $z_1$, $z_2$, ..., $z_K$) to be 1?

enter image description here

| cite | improve this question | | | | |
  • 3
    $\begingroup$ $z_k$ is a binary component of a $K$ dimensional vector, $z$. The purpose of $z$ is to indicate which one of the $K$ populations a particular observation comes from. Under the model, an observational element must belong to one and only one of the $K$ populations. Hence the summation to 1. $\endgroup$ – lmo Nov 25 '16 at 20:39
  • $\begingroup$ Imo is correct but to complete the argument the probability that zk is 1 is the proportion that corresponds to that specific normal distribution in the mixture. $\endgroup$ – Michael R. Chernick Nov 25 '16 at 23:29
  • 2
    $\begingroup$ "binary component" means that it is binary, either 0 or 1, and that it is one element in the vector. It is more or less a combination of two math terms, but probably with a computer science influence. To emphasize, let's say $z = (0, 1, 0, 0, 0)$. Then $z_1=0$, $z_2=1$, and $z_k=0$ for $k \in \{1, 3, 4, 5 \}$. $\endgroup$ – lmo Nov 26 '16 at 13:01
  • 1
    $\begingroup$ That is correct. $\endgroup$ – lmo Nov 27 '16 at 13:42
  • 1
    $\begingroup$ just fyi it's usually called the one-hot representation $\endgroup$ – dontloo Dec 1 '16 at 9:18

For each $x_i$ in the sample $(x_1,\ldots,x_n)$ one can create an binary (and unobserved) vector $\mathbf{z}_i=(z_{i1},\ldots,z_{iK})$ made of components in $\{0,1\}$ such that one and only one of the $z_{ij}$'s is equal to one, e.g., $\mathbf{z}_i=(0,\ldots,0,1,0,\ldots,0)$, and all others are equal to zero. The corresponding $j$ is called the component of $x_i$. Conditional on $\mathbf{z}_i$, $x_i$ is distributed as a normal variate $$\mathcal{N}\left(\sum_{j=1}^K z_{ij}\mu_j,\sum_{j=1}^K z_{ij}\Sigma_j\right)$$Marginally, the vector $\mathbf{z}_i$ is distributed as a Multinomial $\mathcal{M}_K(1;\pi_1,\ldots,\pi_k)$ variate.

| cite | improve this answer | | | | |
  • $\begingroup$ Thanks Xi'an, vote up, what means $\mu_j$ and $\sum_j$ in your formula? $\endgroup$ – Lin Ma Dec 1 '16 at 7:32
  • 1
    $\begingroup$ This is the same notation as in formula (9.7). $\endgroup$ – Xi'an Dec 1 '16 at 8:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.