If $p \sim \Pi$, then $p$ is a random variable and $\Pi$ is a distribution but then how is $X_1,\ldots,X_n \,|\, p \stackrel{i.i.d.}{\sim} p$?

Question

I am trying to understand the following from https://www.dianacai.com/blog/2021/02/14/schwartz-theorem-posterior-consistency/

We consider a model class given by a space of densities $\mathcal{P}$ with respect to a $\sigma$-finite measure $\mu$, and we denote the distribution of a density $p \in \mathcal{P}$ as $P$, i.e., $p = \frac{dP}{d\mu}$. Denote the joint distribution of $n \in \mathbb{N} \cup \{\infty\} $ samples by $P^{(n)}$.

Let $\Pi$ be a prior distribution on our space of models $\mathcal{P}$, consider the following Bayesian model: \begin{align} p \sim \Pi \\ X_1,\ldots,X_n \,|\, p \stackrel{i.i.d.}{\sim} p, \end{align}

If $p \sim \Pi$, then $p$ is a random variable and $\Pi$ is a distribution but then how is $X_1,\ldots,X_n \,|\, p \stackrel{i.i.d.}{\sim} p$?

I mean is it possible to write $x\sim y,y\sim z$. How does this make sense?

Also, is $P$ the cumulative distribution here?

Answer 1

This is a hierarchical model defined by the conditional distribution of the $X_i$'s given $P$ and the marginal distribution of $P$, represented by its density $p$, as $\Pi=\Pi(p)$.

In a finite setting when the $X_i$'s take values in a finite set $\mathfrak X$, for instance $\{1,\ldots,k\}$, $p$ is the probability mass function (pmf) represented by $(\rho_1,\ldots,\rho_k)$ where $$\sum_{j=1}^k \rho_j=1$$ and $X_i\sim p$ means that $$P(X_i=j)=\rho_j$$ Then, $p\sim \Pi$ signifies that the vector $(\rho_1,\ldots,\rho_k)$ is distributed as $\Pi$, which is a distribution on the $k$-dimensional simplex, for instance a Dirichlet distribution. The marginal distribution of the $n$-sample is then $$\int \prod_{i=1}^n \rho_{x_i} \,\text d\Pi(\rho_1,\ldots,\rho_k)$$ and the $X_i$'s are no longer independent (but exchangeable).

In a continuous setting when the $X_i$'s take values in a continuous space like $\mathbb R^k$ and $p$ is, e.g., a density wrt the Lebesgue measure on that space, with no further constraint, the distribution on $p$ is then a probability distribution $\Pi$ over the set $\cal P$ of probability measures, i.e., a a family of stochastic processes whose realizations are probability distributions. An example of such distributions are the Dirichlet processes often found in Bayesian non-parametrics.

Still within a continuous setting, if instead the family $\cal P$ of probability densities is chosen to be parameterised, for instance (when $\mathfrak X=\mathbb R$) the set of all Normal densities, $$\mathcal P=\{\text{N}(\mu,\sigma);\ \mu\in\mathbb R\,,\ \sigma\in\mathbb R^+\}$$ $\Pi$ is (the image of) a probability distribution over $\mathbb R\times\mathbb R^+$. The marginal distribution of the $n$-sample is then $$\int \frac{1}{(2\pi\sigma^2)^{n/2}}\exp\big\{-\sum_{i=1}^n (x_i-\mu)^2/2\big\}\,\text{d}\Pi(\mu,\sigma)$$ and the $X_i$'s are no longer independent (but exchangeable).

Answer 2

In this model, $p$ is the (random) density function for the $X_i$ values and $\Pi$ is the distribution for this random density.