# If $p \sim \Pi$, then $p$ is a random variable and $\Pi$ is a distribution but then how is $X_1,\ldots,X_n \,|\, p \stackrel{i.i.d.}{\sim} p$?

I am trying to understand the following from https://www.dianacai.com/blog/2021/02/14/schwartz-theorem-posterior-consistency/

We consider a model class given by a space of densities $$\mathcal{P}$$ with respect to a $$\sigma$$-finite measure $$\mu$$, and we denote the distribution of a density $$p \in \mathcal{P}$$ as $$P$$, i.e., $$p = \frac{dP}{d\mu}$$. Denote the joint distribution of $$n \in \mathbb{N} \cup \{\infty\}$$ samples by $$P^{(n)}$$.

Let $$\Pi$$ be a prior distribution on our space of models $$\mathcal{P}$$, consider the following Bayesian model: \begin{align} p \sim \Pi \\ X_1,\ldots,X_n \,|\, p \stackrel{i.i.d.}{\sim} p, \end{align}

If $$p \sim \Pi$$, then $$p$$ is a random variable and $$\Pi$$ is a distribution but then how is $$X_1,\ldots,X_n \,|\, p \stackrel{i.i.d.}{\sim} p$$?

I mean is it possible to write $$x\sim y,y\sim z$$. How does this make sense?

Also, is $$P$$ the cumulative distribution here?

• I assume you are familiar with the idea of conditional probability. The notation $|\,p$ in the displayed equation indicates that the equation is conditional on $p$, meaning that $p$ is treated as known for the purpose of the equation. Commented Aug 7, 2022 at 22:12
• The fact that $X_i\sim p$ is simply the definition of $p$. Yes, $p$ is a distribution. Even distributions can be randomly chosen from a set of possibilities. $\Pi$ is simply the set of possible distributions that $p$ might be chosen from. The blog you are reading is using mathematical language that is perhaps more abstract than strictly necessary. If you are not comfortable with this level of mathematical abstraction, there might be other references to read that make the same statements using lower level math. Commented Aug 7, 2022 at 22:47
• No this does not mean that $X_i|p \sim \Pi$. I am unclear why you would think that. The equations mean exactly they say, not something else! Commented Aug 7, 2022 at 22:48
• When the blog author says that "$p:{\cal X} \rightarrow R$ is a measurable function that is nonnegative and integrates to 1", this is just a pure math way of saying that $p$ is a probability density or probability mass function and that ${\cal X}$ is the sample space. In other words, the statement is defining a random variable $X$ with probability density $p$. Commented Aug 7, 2022 at 22:59
• You're forgetting that there are many normal distributions, not just one. If we take $p(x)$ to be $N(\mu,\sigma^2)$ then $\Pi$ would be the family of all possible normal distributions where $\mu$ is any possible real number and $\sigma^2$ is any possible positive number. Commented Aug 7, 2022 at 23:33

This is a hierarchical model defined by the conditional distribution of the $$X_i$$'s given $$P$$ and the marginal distribution of $$P$$, represented by its density $$p$$, as $$\Pi=\Pi(p)$$.

In a finite setting when the $$X_i$$'s take values in a finite set $$\mathfrak X$$, for instance $$\{1,\ldots,k\}$$, $$p$$ is the probability mass function (pmf) represented by $$(\rho_1,\ldots,\rho_k)$$ where $$\sum_{j=1}^k \rho_j=1$$ and $$X_i\sim p$$ means that $$P(X_i=j)=\rho_j$$ Then, $$p\sim \Pi$$ signifies that the vector $$(\rho_1,\ldots,\rho_k)$$ is distributed as $$\Pi$$, which is a distribution on the $$k$$-dimensional simplex, for instance a Dirichlet distribution. The marginal distribution of the $$n$$-sample is then $$\int \prod_{i=1}^n \rho_{x_i} \,\text d\Pi(\rho_1,\ldots,\rho_k)$$ and the $$X_i$$'s are no longer independent (but exchangeable).

In a continuous setting when the $$X_i$$'s take values in a continuous space like $$\mathbb R^k$$ and $$p$$ is, e.g., a density wrt the Lebesgue measure on that space, with no further constraint, the distribution on $$p$$ is then a probability distribution $$\Pi$$ over the set $$\cal P$$ of probability measures, i.e., a a family of stochastic processes whose realizations are probability distributions. An example of such distributions are the Dirichlet processes often found in Bayesian non-parametrics.

Still within a continuous setting, if instead the family $$\cal P$$ of probability densities is chosen to be parameterised, for instance (when $$\mathfrak X=\mathbb R$$) the set of all Normal densities, $$\mathcal P=\{\text{N}(\mu,\sigma);\ \mu\in\mathbb R\,,\ \sigma\in\mathbb R^+\}$$ $$\Pi$$ is (the image of) a probability distribution over $$\mathbb R\times\mathbb R^+$$. The marginal distribution of the $$n$$-sample is then $$\int \frac{1}{(2\pi\sigma^2)^{n/2}}\exp\big\{-\sum_{i=1}^n (x_i-\mu)^2/2\big\}\,\text{d}\Pi(\mu,\sigma)$$ and the $$X_i$$'s are no longer independent (but exchangeable).

• "defined by the conditional distribution of the $X_i$'s given $P$" do you mean given $p$.
– gbd
Commented Aug 8, 2022 at 5:53
• By "the marginal distribution of $P$, represented by $p$ as $\Pi$" do you mean $p$ is the marginal probability of $P$? i.e. we get $p$ by integrating $P$ with respect to $μ$ and $σ$?
– gbd
Commented Aug 8, 2022 at 6:28
• To get the marginal distribution, why do we integrate $P$ with respect to $d\Pi(\mu,\sigma)$ instead of $d\mu d\sigma$ like it is usually done when calculating marginal probabilities.
– gbd
Commented Aug 8, 2022 at 6:46
• The notation $d\Pi(\mu,\sigma)$ means integrating wrt the prior measure on the parameters, there is nothing unusual there. It is a generalisation of $\pi(\mu,\sigma)d(\mu,\sigma)$. Commented Aug 8, 2022 at 8:29
• Is it safe to say that the prior $\pi(\mu,\sigma)=\frac{d\Pi(\mu,\sigma)}{d(\mu,\sigma)}$?
– gbd
Commented Aug 8, 2022 at 9:55

In this model, $$p$$ is the (random) density function for the $$X_i$$ values and $$\Pi$$ is the distribution for this random density.

• but how does $X_1,\ldots,X_n \,|\, p\,{\sim} \,p$ make sense? I mean this is like if we have two random variables $X$ and $Y$ and then we say $X|Y\sim Y$... but $Y$ is random variable not a distribution so how can we write $\sim Y$?
– gbd
Commented Aug 8, 2022 at 10:01
• It's not really dissimilar to a simple model like $X_1,...,X_n|\theta \sim \text{Bern}(\theta)$ with prior $\theta \sim \pi$. The only difference here is that $p$ directly represents the (random) density instead of a parameter indexing that density. The quoted section in your question already states that $p$ is a random density.
– Ben
Commented Aug 8, 2022 at 10:04
• what is the difference between a random density and a density?
– gbd
Commented Aug 8, 2022 at 10:09
• is this like saying $X_1,\ldots,X_n\,|\,\text{Bern}(\theta)\,\sim\,\text{Bern}(\theta)$?
– gbd
Commented Aug 8, 2022 at 10:16
• It's similar, though usually if you have a parameterised density you sould just state conditioning on the parameter. As to the density being random, that is the essence of the Bayesian model, where we give $p$ a prior distribution.
– Ben
Commented Aug 9, 2022 at 1:07