Truncated Dirichlet process vs Dirichlet distribution

Question

As the topic suggests I was wondering what the main differences are in using one over the other. Suppose for sake of simplicity the Dirichlet distribution has all parameters set to $\alpha$.

All I can think of is that considering that DPs use a stick breaking process the first few sticks tend to be larger. Since the last few sticks are smaller the clusters associated to them tend to be unoccupied. Thus the DP will limit the number of clusters available.

However it's conceivable for a given number of clusters $T$ both a truncated DP and a Dirichlet distribution will give the same realisation of $\pi$s.

For illustration of what I mean refer to the following two generative processes:

1. Dirichlet Distribution: $$ \begin{align} \theta_i&\sim\mathcal{N}(0,100) \quad i=1,\cdots,T\\ \mathbf{\pi}&\sim Dir(\alpha)\\ z_n&\sim\prod_{i=1}^T \pi_i^{1(z_n=i)}\\ x_n&\sim \mathcal{N}(\theta_i,1) \end{align} $$

2. Dirichlet Process: $$ \begin{align} \theta_i&\sim\mathcal{N}(0,100) \quad i=1,\cdots,T\\ v_i&\sim Be(1,\alpha) \quad i=1,\cdots,T\\ \pi_i & = v_i\prod_{j=1}^{i-1}(1-v_j)\\ z_n&\sim\prod_{i=1}^T \pi_i^{1(z_n=i)}\\ x_n&\sim \mathcal{N}(\theta_i,1) \end{align} $$

I understand that the same $\alpha$ may not derive the same $\pi_i$s but the point is in the case of the Dirichlet distribution,for an appropriate $\alpha$, won't (some of the $\pi_i$s be driven to zero) so that we get the same posterior on $\pi_i$?

Answer 1

You have two very different priors on $\pi$, so why would you expect the same posterior?

Let's think about the nature of $T$ in your presentation above:

• In 1: Dirichlet Distribution, $T$ is a fundamental defining feature of the finite-dimensional Dirichlet prior distribution for $\mathbf{\pi}$. With $T$ fixed, you have complete flexibility (within the unit simplex constraint) to set the values of $\alpha$. Without any additional structure favoring some values of $\alpha$ over others, as $T$ increases, the expected value of each element of $\mathbf{\pi}$ will decrease.

• In 2: Dirichlet Process, $T$ is a choice of truncation level, not a feature or parameter of the ideal Dirichlet Process. In this case, as $T$ gets large, the expected values for the individual elements of $\pi$ do not shrink, at least not for the portion of the process you're approximating well. As you noted above, the first few elements of $\pi$ are still large in expectation due to the stick-breaking construction. Your hope is that $T$ is large enough to provide an adequate approximation to the infinite-dimension process that you really wanted to sample from.