6
$\begingroup$

I'm trying to understand the construction process of DP, however, with little background in measure theory, the original papers are hard to read, but I believe the ideas behind these papers can be followed.

Let $y_1, y_2, \ldots, y_n$ be i.i.d. with distribution function $F$, which is characterized by $k$ unknown probabilities $\theta_1, \ldots, \theta_k$, i.e., $y_i$s are discrete, taking a finite number $k$ of possible values. Bayesian inference based on these observations often put a Dirichlet prior on the parameters of $F$. Posterior and predictive distribution can then be calculated. That finite mixture models is not hard to understand, but...

When the $y_i$s can take infinitely many possible values ($k \rightarrow \infty$), then the Bayesian nonparametric models can deal with this situation but I don't understand why. Here are my questions:

  1. $k \rightarrow \infty$ means that $y_i$ can take any point mass on $\mathcal{R}$, so it is a continuous r.v. isn't it?
  2. In the finite case, $F$ is indexed by $1, \ldots, k$, while in the infinite case, $F$ should be indexed by partitions $(T_1, \ldots, T_k)$ (Ferguson, 1972), how can I get the intuition of the latter case from the former?
$\endgroup$
1
$\begingroup$

1) infinite does not mean continuous. 2) The total mass (measure) of a probabilistic measure F is always 1. In the former case, F is partitioned by k bins (discrete). However F is not required to be discrete. If F is continuous, it can be partitioned by k regions (T1...Tk)

$\endgroup$
1
$\begingroup$

Not an answer but rather a long comment regarding your point 2 (I am not an expert so take my explanation with caution)

It was not clear to me in what sense you have used the terms infinite and finite in your statement 2. Below is my take on how you can argue that indeed "In the finite case, $F$ is indexed by $1,\ldots,k$ while in the infinite case, $F$ should be indexed by partitions...", the later I think should be changed to indexed by sets not partitions.

$\underline{Finite~~case}$

When you say "in the finite case" you probably refer to the situation where you take $$F \sim \text{Dirichlet } \textbf{distribution } \text{of dimension } k\,.$$

That is, your $F$ is a random vector in $\mathbb{R}^k$ drawn according to the Dirichlet distribution. In particular this means that $ F \equiv(x_1, \ldots x_k)$ admits $\sum_i^k x_i = 1$. The fact that elements of $F$ sum to $1$ allows you to associate with $F$, so to speak, a distribution $\widetilde{F}$ over indexes $i= 1,\ldots, k$, indeed take $$\widetilde{F}(m) := \sum_{j \leq m} x_i \,.$$ This $\widetilde{F}$ is a valid distribution on indexes $1, \ldots, k$. Namely you can define random variable $Z$ on $1, \ldots, k$ which is distributed according to $\widetilde{F}$ meaning that

$$ P(X \leq m) = \widetilde{F}(m) := \sum_{j \leq m} x_i\,.$$

The scheme therefor allows you to generate a random distribution over index set $\{1, \ldots, k\}$ (random since $\{x_i\}$ and therefor your $\widetilde{F}$ is random).

$\underline{Infinite~~case}$.

In the infinite case, I presume you refer to $\widetilde{F}$ drawn from Dirichle process in that case $\widetilde{F}$ is already a distribution, no need to associate a new distribution with it. So Dirichle process is already a scheme to generate random distributions. This random $\widetilde{F}$ is defined on measure space $\Omega$, where $\Omega$ is in turn the domain of "base distribution" $H_0:\Omega \to \mathbb{R}$ which is a parameter of a Dirichle process .

Ultimatly, $\widetilde{F}$ defines a random variable $Z$ on $\Omega$, which is distributed for $A \subset \Omega$ according to

$$P(Z \in A) \sim \widetilde{F}$$

unlike in the finite case where our random variable was defined on finite index set here $Z$ is defined on any subset of sigma algebra of $\Omega$.

Lastly to draw the analogy with Dirichlet distribution property that $\sum_{i=1}^k x_i = 1$, note that we must have for any (disjoint) partition $\Omega = \cup_{i=1}^k A_i$ that $\sum_{i=1}^k P(Z \in A_i) = 1$. Here $P(Z \in A_i)$ take the place of $x_i$, however we can take $k$ to be $\infty$, meaning a partition $\cup_{i=1}^\infty A_i$ and still $\sum_{i=1}^\infty P(Z \in A_i) = 1$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.