3
$\begingroup$

I have been reading a ton of stuff online and have a question about my understanding of Dirichlet Process Mixtures. First some basics on what I understand:

Dirichlet Distribution: multivariate generalization of beta distribution

Probability measure: function that assigns subsets to values in [0,1] (a probability)

Dirichlet Process (DP): $P$, a probability measure, is a DP if for every finite disjoint partition $S_1,S_2,\dots S_n$ of space $\omega$, the following is true:

($P(S_1), P(S_2),\dots P(S_n)$) has Dirichlet Distribution with parameters $\alpha *P_0(S_1), \alpha * P_0(S_2),\dots\alpha * P_0(S_n)$

where $E(P(S_i)) = P_0$

Ok, I get it so far. Let's move onto Dirichlet Process MIXTURES

Dirichlet Process Mixture = hierarchical model where

  1. $x_i | \theta_i$ has distribution $f_\theta$

  2. $\theta_1, \theta_2\dots\theta_n$ has distribution $P$

  3. $P$ is a Dirichlet Process with params $\alpha$ and $P_0$

So let me check to see if I understand these 3 lines correctly

What first line is saying:

a point $x$ comes from a distribution, let's say $f_\theta$ = normal with parameters $\mu = 0$ and $\sigma = 1$

What the second and third line are saying is:

My space omega is partitioned into parameter subsets. So let's say our components are normals. So $\theta_1$ = mean 0 with std 1, $\theta_2$ = mean 3 with std 5, ...

So in essence $\omega = \theta_1 \cup \theta_2 \cup \dots$

So $P(\theta_i)$ = probability of us getting the parameter set $i$

Since $P$ is a DP,

$(P(\theta_1), P(\theta_2)\dots)$ has distribution $\mathrm{Dir}(\alpha * P_0(\theta_1), \alpha * P_0(\theta_2)\dots)$

Am I correct?

$\endgroup$
2
$\begingroup$

Probably it is better to describe how one would generate data from a Dirichlet process mixture. Each line is understood to be conditional on all lines above it.

  1. Sample $P \sim \operatorname{DP}(\alpha P_0)$.
  2. Sample $\theta_1, \ldots, \theta_n \sim P$ independently.
  3. Sample $X_1, \ldots, X_n$ independently such that $X_i \sim f_{\theta_i}$.

That's all there is to it. The only thing different from drawing from a DP versus a DP-mixture is that if you are just drawing data from a DP you stop after step 2 and take the $\theta_i$'s as your data. For a DP mixture, the $\theta_i$'s are not observed but instead represent latent variables.

$\endgroup$
  • $\begingroup$ But do the thetas in the Dirichlet Process Mixture definition correspond to the S's in the Dirichlet Process definition? $\endgroup$ – ilikecats May 28 '18 at 7:17
  • $\begingroup$ @ilikecats No, they do not. $\endgroup$ – guy May 28 '18 at 15:52
  • $\begingroup$ Then what does? $\endgroup$ – ilikecats May 28 '18 at 16:07
  • $\begingroup$ The reason I thought it would be the thetas is bc of this link: docs.pymc.io/notebooks/dp_mix.html which says that P0 is a probability measure on the parameter space. That makes me think the thetas are like the S's $\endgroup$ – ilikecats May 28 '18 at 16:15
  • $\begingroup$ @ilikecats the $S$’s would correspond to subsets of the space of $\theta$’s. $\endgroup$ – guy May 28 '18 at 18:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.