Understanding Dirichlet Process Mixtures

Question

I have been reading a ton of stuff online and have a question about my understanding of Dirichlet Process Mixtures. First some basics on what I understand:

Dirichlet Distribution: multivariate generalization of beta distribution

Probability measure: function that assigns subsets to values in [0,1] (a probability)

Dirichlet Process (DP): $P$, a probability measure, is a DP if for every finite disjoint partition $S_1,S_2,\dots S_n$ of space $\omega$, the following is true:

($P(S_1), P(S_2),\dots P(S_n)$) has Dirichlet Distribution with parameters $\alpha *P_0(S_1), \alpha * P_0(S_2),\dots\alpha * P_0(S_n)$

where $E(P(S_i)) = P_0$

Ok, I get it so far. Let's move onto Dirichlet Process MIXTURES

Dirichlet Process Mixture = hierarchical model where

1. $x_i | \theta_i$ has distribution $f_\theta$

2. $\theta_1, \theta_2\dots\theta_n$ has distribution $P$

3. $P$ is a Dirichlet Process with params $\alpha$ and $P_0$

So let me check to see if I understand these 3 lines correctly

What first line is saying:

a point $x$ comes from a distribution, let's say $f_\theta$ = normal with parameters $\mu = 0$ and $\sigma = 1$

What the second and third line are saying is:

My space omega is partitioned into parameter subsets. So let's say our components are normals. So $\theta_1$ = mean 0 with std 1, $\theta_2$ = mean 3 with std 5, ...

So in essence $\omega = \theta_1 \cup \theta_2 \cup \dots$

So $P(\theta_i)$ = probability of us getting the parameter set $i$

Since $P$ is a DP,

$(P(\theta_1), P(\theta_2)\dots)$ has distribution $\mathrm{Dir}(\alpha * P_0(\theta_1), \alpha * P_0(\theta_2)\dots)$

Am I correct?

Answer 1

Probably it is better to describe how one would generate data from a Dirichlet process mixture. Each line is understood to be conditional on all lines above it.

1. Sample $P \sim \operatorname{DP}(\alpha P_0)$.
2. Sample $\theta_1, \ldots, \theta_n \sim P$ independently.
3. Sample $X_1, \ldots, X_n$ independently such that $X_i \sim f_{\theta_i}$.

That's all there is to it. The only thing different from drawing from a DP versus a DP-mixture is that if you are just drawing data from a DP you stop after step 2 and take the $\theta_i$'s as your data. For a DP mixture, the $\theta_i$'s are not observed but instead represent latent variables.