Drawing from the joint distribution by drawing from the marginal and conditional distribution

Question

How can we prove that a draw $(x,y)$ from the joint probability distribution of two random variables $X$ and $Y$ can be obtained by first generating a draw $x$ from the marginal probability distribution of $X$ and then a draw $y$ from the conditional probability distribution of $Y$ given $X=x$?
Is this a simple consequence of the definition of a conditional probability distribution?

Answer 1

In short, drawing from the joint distribution is equivalent to first drawing from one marginal and second from the corresponding conditional.

To answer a side question from @statmerkur in the comments, drawing from a distribution $F$ means producing a realisation of a random variable with distribution $F$, either by inducing the phenomenon distributed as $F$ (e.g., the time to an electron emission) or by simulating it (e.g., taking $F^{-1}(U)$ with $U$ a uniform variate when the distribution is univariate).

Considering that$$\mathbb P(X\in A,\ Y\in B)=\mathbb P(X\in A)\mathbb P(Y\in B|X\in A)$$ $-$where $\mathbb P(\cdot)$ denotes the probability of the event between parentheses$-$, we have \begin{align} \mathbb P\{X\in (\epsilon,\epsilon+\text d\epsilon),\ Y\in (\eta,\eta+\text d\eta)\}&=\int_{\epsilon}^{\epsilon+\text d\epsilon} \int_{\eta}^{\eta+\text d\eta} f_{Y|X}(y|x)\,\text dy\,f_X(x)\,\text dx\\ &=\mathbb E^X[\mathbb I_{(\epsilon,\epsilon+\text d\epsilon)}(X)\times\mathbb E^{Y|X}\{\mathbb I_{(\eta,\eta+\text d\eta)}(Y)|X\}] \end{align} where [in reply to @singlemalt]

1. $f_{Y|X}(\cdot|\cdot)$ denotes the probability density of the conditional distribution of $Y$ given $X$
2. $\mathbb I_A(X)$ is the indicator function that $X\in A$, equal to either 0 or 1, with $$\mathbb P(X\in A)=\mathbb E^X[\mathbb I_A(X)]$$
3. $\mathbb E$ is a blackboard-bold "E" that is often used for representing `expectations'
4. the superscript $X$ is indicating that the expectation is wrt the distribution of the rv $X$,
5. and the superscript $Y|X$ is indicating that the expectation is computed wrt the conditional distribution of the rv $Y$ given the rv $X$, meaning that $X$ is considered as fixed when computing this conditional expectation.

Therefore, if [one simulates] $X\sim f_X(x)$, with realisation $x$, and [then one simulates] $Y\sim f_Y|X(y|x)$, $$\mathbb I_{(\epsilon,\epsilon+\text d\epsilon)}(X)\times\mathbb I_{(\eta,\eta+\text d\eta)}(Y)$$ is an unbiased estimator of $$\mathbb E^{(X,Y)}[\mathbb I_{(\epsilon,\epsilon+\text d\epsilon)}(X)\times\mathbb I_{(\eta,\eta+\text d\eta)}(Y)]=\mathbb E^X[\mathbb I_{(\epsilon,\epsilon+\text d\epsilon)}(X)\times\mathbb E^{Y|X}\{\mathbb I_{(\eta,\eta+\text d\eta)}(Y)|X\}]$$ for all $(\epsilon,\eta)$. Letting $\text d\epsilon$ and $\text d\eta$ go to zero (0) allows one to conclude that the joint distribution of $(X,Y)$ is obtained through this decomposition marginal-then-conditional, thus that the simulation is producing a realisation from the correct (joint) distribution.

Take the example of a bivariate Normal $$(X,Y)^\prime\sim\mathcal N_2(0_2,\Sigma)\qquad \Sigma=\left(\begin{matrix}1 &\rho\cr\rho &1\end{matrix} \right)\qquad \rho\in(-1,1)$$ Then drawing $X$ from the marginal $\mathcal N(0,1)$ is equivalent to setting $X=\epsilon_x$ with $\epsilon_x\sim \mathcal N(0,1)$. Further, drawing $Y$ from the conditional $\mathcal N(\rho x,1-\rho^2)$ is equivalent to setting $$Y=\rho x +\sqrt{1-\rho^2}\epsilon_y\qquad\epsilon_y\sim\mathcal N(0,1)$$
Therefore $$ \left(\begin{matrix}X\cr Y\cr\end{matrix} \right)= \overbrace{\left(\begin{matrix}1 &0\cr \rho &\sqrt{1-\rho^2}\cr\end{matrix} \right)}^A \left(\begin{matrix}\epsilon_x\cr \epsilon_y\cr\end{matrix} \right) \sim \mathcal N_2(0_2,\Sigma)$$ since $$A A^\mathsf{T}=\Sigma$$ This implies that the draw is truly from the joint distribution.