3
$\begingroup$

As the topic suggests I was wondering what the main differences are in using one over the other. Suppose for sake of simplicity the Dirichlet distribution has all parameters set to $\alpha$.

All I can think of is that considering that DPs use a stick breaking process the first few sticks tend to be larger. Since the last few sticks are smaller the clusters associated to them tend to be unoccupied. Thus the DP will limit the number of clusters available.

However it's conceivable for a given number of clusters $T$ both a truncated DP and a Dirichlet distribution will give the same realisation of $\pi$s.

For illustration of what I mean refer to the following two generative processes:

1. Dirichlet Distribution: $$ \begin{align} \theta_i&\sim\mathcal{N}(0,100) \quad i=1,\cdots,T\\ \mathbf{\pi}&\sim Dir(\alpha)\\ z_n&\sim\prod_{i=1}^T \pi_i^{1(z_n=i)}\\ x_n&\sim \mathcal{N}(\theta_i,1) \end{align} $$

2. Dirichlet Process: $$ \begin{align} \theta_i&\sim\mathcal{N}(0,100) \quad i=1,\cdots,T\\ v_i&\sim Be(1,\alpha) \quad i=1,\cdots,T\\ \pi_i & = v_i\prod_{j=1}^{i-1}(1-v_j)\\ z_n&\sim\prod_{i=1}^T \pi_i^{1(z_n=i)}\\ x_n&\sim \mathcal{N}(\theta_i,1) \end{align} $$

I understand that the same $\alpha$ may not derive the same $\pi_i$s but the point is in the case of the Dirichlet distribution,for an appropriate $\alpha$, won't (some of the $\pi_i$s be driven to zero) so that we get the same posterior on $\pi_i$?

$\endgroup$
  • $\begingroup$ Quite old question, but have you found an answer here? $\endgroup$ – fsociety Apr 29 '16 at 11:18
  • $\begingroup$ @fsociety not a full answer but from experience (in using variational DPs) DPs tend to assign smaller number of clusters. Whereas normal Dirichlet distributions will tend to fill up the entire set. $\endgroup$ – sachinruk Apr 29 '16 at 19:56
  • $\begingroup$ Very old question, but a relevent link to a paper by Hemant Ishwaran, who did a lot of work on these models: biomet.oxfordjournals.org/content/87/2/371.full.pdf $\endgroup$ – HStamper Mar 31 '17 at 19:29
4
$\begingroup$

You have two very different priors on $\pi$, so why would you expect the same posterior?

Let's think about the nature of $T$ in your presentation above:

  • In 1: Dirichlet Distribution, $T$ is a fundamental defining feature of the finite-dimensional Dirichlet prior distribution for $\mathbf{\pi}$. With $T$ fixed, you have complete flexibility (within the unit simplex constraint) to set the values of $\alpha$. Without any additional structure favoring some values of $\alpha$ over others, as $T$ increases, the expected value of each element of $\mathbf{\pi}$ will decrease.

  • In 2: Dirichlet Process, $T$ is a choice of truncation level, not a feature or parameter of the ideal Dirichlet Process. In this case, as $T$ gets large, the expected values for the individual elements of $\pi$ do not shrink, at least not for the portion of the process you're approximating well. As you noted above, the first few elements of $\pi$ are still large in expectation due to the stick-breaking construction. Your hope is that $T$ is large enough to provide an adequate approximation to the infinite-dimension process that you really wanted to sample from.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.