Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
So far I only really understand the Dirichlet Process through its various metaphors. For the Polya Urn scheme, my understanding is that the "base distribution" is the original distribution of colors in the urn before you start the process. So if alpha is very large then somehow your population ends up resembling the base distribution.
But that's as far as my understanding goes. How does the base distribution affect the chinese restaurant process? Or the dirichlet process in general?
Let $$ G \sim \textsf{DP}(\alpha, H) $$ which says that the random distribution $G$ is itself distributed according to the Dirichlet Process with concentration parameter $\alpha$ and base distribution $H$. There is an explicit representation for $G$ and it's useful for understanding the role of the base distribution and its relation to "clustering" that goes on. In particular, $G$ is a discrete distribution with random support points and random weights: $$ G = \sum_{c=1}^\infty w_c\, \delta_{\theta_c} , $$ where $\delta_x$ is a point-mass located at $x$ and $\sum_{c=1}^\infty w_c = 1$. The distributions for the component weights $w = (w_1, w_2, \ldots)$ and corresponding component parameters $\theta = (\theta_1, \theta_2, \ldots)$ are given by \begin{align} w &\sim \textsf{Stick}(\alpha) \\ \theta_c &\stackrel{\text{iid}}{\sim} H . \end{align} The stick-breaking weights are generated according to $$ w_c = v_c \prod_{\ell = 1}^{c-1} (1 - v_\ell) \qquad\text{where $v_c \stackrel{\text{iid}}{\sim} \textsf{Beta}(1,\alpha)$} . $$ The base distribution determines the locations of the support points while the stick-breaking weights determine the amount of clustering.
In the limit as $\alpha \to 0$, the first weight approaches unity: $w_1 \to 1$, in which case the random distribution $G$ has a single support point. In this case, any draw $G$ is quite different from the base distribution $H$. Going the other way, as $\alpha \to \infty$, no finite collection of weights dominate and each random draw of $G$ becomes arbitrarily close to $H$ (i.e., becomes concentrated on $H$).
It is possible to introduce classifications that indicate cluster assignments and integrate out the weights, leaving one with the Chinese Restaurant Process to make the (table) assignments. The base distribution then is used to determine the entrees for the tables.
