Finite mixture models - Basic understanding

Question

I have been reading lecture slides about Dirichlet Process.

In page 22, there is a picture about the following finite mixture model.

$$\phi _{k}\sim H\\ \pi \sim Dirichlet(\alpha /K,\dots,\alpha /K)\\ Z_{i}\rvert\pi \sim Discrete(\pi )\\ x_{i}\rvert\phi _{z_{i}}\sim F(\cdot \vert \phi _{z_{i}})$$

I know the meaning of the following variables (Please correct me if I were wrong):

$N:$ Number of parameters/observations

$K:$ Number of mixture models

$\alpha:$ Dirichlet parameter

$\pi:$ Probability distribution on N variables

$x_{i}$ observed variables or data that we want to model

$H:$ Hyper-parameters, prior distribution about the K mixture models

$\phi _{k}:$ Parameters of the k-th model

$F(\cdot \vert \phi _{z_{i}})$：Individual mixture models

My questions are:

1. What does $z_{i}$ mean and what is the relation with $\pi$ and $x_{i}$?

2. What does $Z_{i}\rvert\pi \sim Discrete(\pi )$ mean?

3. What does $x_{i}\rvert\phi _{z_{i}}\sim F(\cdot \vert \phi _{z_{i}})$ mean?

Thanks!

Answer 1

The Wikipedia site for Dirichlet Latent Allocation contains a brief nice description of the different parameters. $Discrete(\pi)$ stands for a multinomial distribution. It is meant any distribution that allows you to model the fact that each word of a document can be assigned to a topic from a finite set of topics.

Answer 2

At first a brief introduction: in the finite mixture models (HMM) you observe a time series $x_i, t =1,\dots, T$ and you suppose that at each time $t$ you are in a specific regime/state that are represented by the variable $z_i \in \mathbb{Z}^+$. The observations $x_i$ are distributed accordingly to a distribution $F(\cdot)$, but the parameter of the distribution depends on the state you are, so in if you are in state 1 ($z_i=1$) then $x_i \sim F(\phi_i) $ or equivalently $x_i \sim F(\phi_{z_i})$. Regarding the distribution of $Z_i$ i think there is a problem, in the standard HMM the variable assumed by $z_i$ depends on the value of $z_{i-1}$. Let $\boldsymbol{\pi}_i = (\pi_{i1}, \pi_{i2}...)$ be a vector of probability where $pi_{ij}$ mean the probability of be in the state $i$ and in the next time being in the state j, then the probability of $z_i \sim \boldsymbol{\pi}_{z_{i-1}}$

Now:

1) $z_i$ indicates the state on time $i$, it depends on the vector of probability $\pi_{z_{i-1}}$ and decide which parameter use in the distribution of $x_i$

2) is the distribution (multinomial) of $z_i$

3)is the distribution of $x_i$

Answer 3

In other word, associated to each $x_i$ is a random variable $z_i$. $z_i$ is interpreted as an indicator variable, indicating to which component of the mixture the observation $x_i$ belongs. $z_i=k$ imply observation $x_i$ is drawn from component $k$.

Note that $P(z_i=k)=\pi_k$.

Answer 4

$Z_{i}$ is the random variable and ($z_{i}$ is a value it can take) representing the random variable producing $x_{i}$. Interpret $Z_{i}|\pi$ as $P(Z_{i}=z_{i}|\pi)$ and F(.|$\phi(z_{i})$ (this does not match up with your diagram) can be thought of as F(X|Z)