5
$\begingroup$

I know from my measure theory class that for two $\sigma$-finite measure spaces $(\mathcal{X}_1, \mathcal{A}_1, \mu_1)$ and $(\mathcal{X}_2, \mathcal{A}_2, \mu_2)$ there exists a unique measure $\mu := \mu_1 \otimes \mu_2:\mathcal{A}_1 \otimes \mathcal{A}_2 \rightarrow [0, \infty]$ such that $$ \mu_1 \otimes \mu_2(A_1 \times A_2) = \mu_1(A_1)\cdot \mu_2(A_2).$$ So my questions is what that means in probability theory, for the joint distribution (measure) of two random variables? Is $\mathbb{P}_{X, Y}$, the joint distribution of two random variables $X$ and $Y$ the same as $\mathbb{P}_{X}\otimes \mathbb{P}_{Y}$? Probably not, otherwise all random variables would be independent due to the above theorem, no? But how do $\mathbb{P}_{X, Y}$ and $\mathbb{P}_{X}\otimes \mathbb{P}_{Y}$ then relate to each other? (This is particularly needed to compute the expectation over the joint distribution)

$\endgroup$
12
  • 2
    $\begingroup$ The unique measure your theorem says exists with that multiplicative property is the precisely that resulting for the two random variables being independent with respective measures $\mu_1$ and $\mu_2$. Any other measure would not have that multiplicative property for all $A_1 \times A_2 \in \mathcal{A}_1 \otimes \mathcal{A}_2$ $\endgroup$
    – Henry
    May 21, 2021 at 16:44
  • 1
    $\begingroup$ This independence leads (in one direction) to $\mathbb E_{X,Y}[X\cdot Y] = \mathbb E_{X}[X] \cdot \mathbb E_{Y}[Y]$ so long as all the expectations are finite $\endgroup$
    – Henry
    May 21, 2021 at 16:48
  • $\begingroup$ Exactly, but in general random variables are not independent. So why wouldnt this theorem apply to them? $\endgroup$
    – guest1
    May 21, 2021 at 16:49
  • 1
    $\begingroup$ How exactly would it apply? All it would end up showing is that you have found two measures on the product: the original $\mathbb{P}_{X,Y}$ and the one constructed by this theorem. The theorem doesn't assert all joint measures are the same! $\endgroup$
    – whuber
    May 21, 2021 at 17:17
  • $\begingroup$ Well I thought that since the measure from the theorem is unique, it is the only measure on the product sigma algebra. And if we calculate a double integral with the theorem of Fubini, we always use the product measure from the theorem above, no? So if we calculate an expectation value then I would assume that we always use the product measure of the theorem above... $\endgroup$
    – guest1
    May 27, 2021 at 19:24

1 Answer 1

9
$\begingroup$

Joint Distributions and Expectation

In general, the joint distribution of random variables $X$ and $Y$, defined on a common probability space $(\Omega, \mathcal{A}, \mathbb{P})$ and taking values in measurable spaces $(\mathcal{X}, \mathcal{B})$ and $(\mathcal{Y}, \mathcal{C})$, respectively, is the probability measure defined on $(\mathcal{X} \times \mathcal{Y}, \mathcal{B} \otimes \mathcal{C})$ by $$ \mathbb{P}_{X, Y}(E) = \mathbb{P}((X, Y) \in E) $$ for all $E \in \mathcal{B} \otimes \mathcal{C}$.

This is the same as the ordinary distribution of $(X, Y) : \Omega \to \mathcal{X} \times \mathcal{Y}$ when viewed as a single random variable defined on $\Omega$.

Also, $X$ and $Y$ are said to be independent if it holds that $$ \mathbb{P}(X \in B, Y \in C) = \mathbb{P}(X \in B) \mathbb{P}(Y \in C) $$ for all $B \in \mathcal{B}$ and $C \in \mathcal{C}$.

The independence condition can be rephrased in terms of the joint distribution of $X$ and $Y$: $X$ and $Y$ are independent if and only if $$ \mathbb{P}_{X,Y}(B \times C) = \mathbb{P}_X(B) \mathbb{P}_Y(C) $$ for all $B \in \mathcal{B}$ and $C \in \mathcal{C}$. That is, if and only if $$ \mathbb{P}_{X, Y} = \mathbb{P}_X \otimes \mathbb{P}_Y. $$ Thus, the joint distribution of $X$ and $Y$ is the product measure of the (marginal) distributions of $X$ and $Y$ precisely in the case that $X$ and $Y$ are independent. If $X$ and $Y$ are dependent, then their joint distribution is not the product measure of the marginal distributions.

Computing Expectations over Joint Distributions

If $X$ and $Y$, as above, are independent and $f : \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$ is a measurable function (satisfying either non-negativity or integrability with respect to $\mathbb{P}_{X,Y}$), then Fubini's theorem allows you to compute $$ \begin{aligned} \mathbb{E}[f(X, Y)] &= \int_\Omega f(X(\omega), Y(\omega)) \, \mathbb{P}(d\omega) && \text{(def. of expectation)} \\ &= \int_{\mathcal{X} \times \mathcal{Y}} f(x, y) \, \mathbb{P}_{X, Y}(d(x, y)) && \text{(change of variables)} \\ &= \int_{\mathcal{X} \times \mathcal{Y}} f(x, y) \, \mathbb{P}_X\otimes\mathbb{P}_Y(d(x, y)) &&\text{(independence)} \\ &= \int_{\mathcal{Y}} \left(\int_{\mathcal{X}} f(x, y) \, \mathbb{P}_X(dx)\right) \, \mathbb{P}_Y(dy) &&\text{(Fubini's theorem)} \end{aligned} $$ However, if $X$ are $Y$ are not independent, then this argument won't work. Instead, if you want to break an expectation of $f(X,Y)$ into an integral over $\mathcal{X}$ followed by an integral over $\mathcal{Y}$, as we did above, you need to know something about the conditional distribution of $X$ given $Y$.

For what follows, suppose $(\mathcal{X},\mathcal{B})$ and $(\mathcal{Y}, \mathcal{C})$ are "sufficiently nice" measurable spaces, meaning that they admit conditional distributions (this will happen for most spaces in practice; a sufficient condition is being standard Borel).

Then if $\mathbb{P}_{X\mid Y} : \mathcal{B} \times \mathcal{Y} \to [0, 1]$ is a version of the conditional distribution of $X$ given $Y$, then we can proceed similarly to the calculations above: $$ \begin{aligned} \mathbb{E}[f(X, Y)] &= \int_{\mathcal{X} \times \mathcal{Y}} f(x, y) \, \mathbb{P}_{X, Y}(d(x, y)) \\ &= \int_{\mathcal{Y}} \left(\int_{\mathcal{X}} f(x, y) \, \mathbb{P}_{X\mid Y}(dx, y)\right) \, \mathbb{P}_Y(dy) \\ &= \mathbb{E}[\mathbb{E}[f(X, Y) \mid Y]] \end{aligned} $$ (in fact, the formula $\mathbb{E}[f(X, Y)] = \mathbb{E}[\mathbb{E}[f(X, Y) \mid Y]]$ holds even without considering conditional distributions (proof), but it's arguably harder to compute in that case).

If $X$ and $Y$ are independent, then it happens that $\mathbb{P}_{X\mid Y}(B, y) = \mathbb{P}_X(B)$ for every $B \in \mathcal{B}$ and $\mathbb{P}_Y$-almost every $y \in Y$. In this case, the calculation reduces to the first computation above.

In practice, the conditional distribution $\mathbb{P}_{X\mid Y}$ will usually be given by a conditional density $p_{X\mid Y} : \mathcal{X} \times \mathcal{Y} \to [0, \infty)$ of $X$ given $Y$ with respect to some dominating measure $\mu$ on $(\mathcal{X}, \mathcal{B})$, yielding $$ E[f(X, Y)] = \int_{\mathcal{Y}} \left(\int_{\mathcal{X}} f(x, y) p_{X \mid Y}(x, y) \, \mu(dx)\right) \, \mathbb{P}_Y(d y). $$

$\endgroup$
28
  • 1
    $\begingroup$ @guest1 You're right, I fixed that typo. Regarding your second question, the uniqueness of the product measure $\mu \otimes \nu$ is with regard to the condition $(\mu \otimes \nu)(A \times B) = \mu(A)\nu(B)$. That is, there's only one measure satisfying that equation (under conditions like $\sigma$-finiteness), but there can be many others that don't satisfy it. Also, there are versions of Fubini's theorem that hold for non-product measures (e.g., disintegration, which is related to conditional probabilities) $\endgroup$ May 27, 2021 at 19:34
  • 1
    $\begingroup$ @guest1 a sufficient condition on the underlying measurable spaces is that they are standard Borel: a measurable space $(\mathcal{X}, \mathcal{B})$ is standard Borel if there exists a metric $d$ on $\mathcal{X}$ such that $(\mathcal{X}, d)$ is a complete, separable metric space with Borel $\sigma$-algebra $\mathcal{B}$ $\endgroup$ Jun 1, 2021 at 15:28
  • 1
    $\begingroup$ @guest1 Thanks, $\mathbb{P}(dy)$ was a typo; it's fixed. I'm not sure what you mean by "non measure-theoretic way". In practice you compute the expected value of a function of two random variables by conditioning on one of them: $E[f(X,Y)]=E[E[f(X, Y) | Y]]$. The calculation usually reduces to a concrete instance of the last equation in my answer. This works whether or not the variables are independent (it's just redundant if you know beforehand that they're independent). Lastly, the dominating measure $\mu$ in probability theory isn't necessarily Lebesgue measure, but it often is. $\endgroup$ Jun 2, 2021 at 16:31
  • 1
    $\begingroup$ @guest1 two things are happening there: the definition of "density", and Fubini's theorem. If $p$ is a density of $P$ with respect to a measure $\mu$, then $\int f\,dP=\int fp\,d\mu$. Next, if the integral exists and is over a product space $\mathcal{X}\times\mathcal{Y}$ and the dominating measure $\mu$ is a product measure $\nu\otimes\lambda$ (as is the case for Lebesgue measure), then the integral can be written $\int_{\mathcal{X}\times\mathcal{Y}}fp\,d(\nu\otimes\lambda)=\int_{\mathcal{Y}}\left(\int_{\mathcal{X}}f(x,y)p(x,y)\,\mu(dx)\right)\,\lambda(dy)$ by Fubini's theorem. $\endgroup$ Jun 3, 2021 at 16:35
  • 1
    $\begingroup$ @guest1 that’s correct $\endgroup$ Jun 4, 2021 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.