Joint probability measure I know from my measure theory class that for two $\sigma$-finite measure spaces $(\mathcal{X}_1, \mathcal{A}_1, \mu_1)$ and $(\mathcal{X}_2, \mathcal{A}_2, \mu_2)$ there exists a unique measure $\mu := \mu_1 \otimes \mu_2:\mathcal{A}_1 \otimes \mathcal{A}_2 \rightarrow [0, \infty]$ such that
$$ \mu_1 \otimes \mu_2(A_1 \times A_2) = \mu_1(A_1)\cdot \mu_2(A_2).$$
So my questions is what that means in probability theory, for the joint distribution (measure) of two random variables?
Is $\mathbb{P}_{X, Y}$, the joint distribution of two random variables $X$ and $Y$ the same as $\mathbb{P}_{X}\otimes \mathbb{P}_{Y}$? Probably not, otherwise all random variables would be independent due to the above theorem, no?
But how do $\mathbb{P}_{X, Y}$ and $\mathbb{P}_{X}\otimes \mathbb{P}_{Y}$ then relate to each other? (This is particularly needed to compute the expectation over the joint distribution)
 A: Joint Distributions and Expectation
In general, the joint distribution of random variables $X$ and $Y$, defined on a common probability space $(\Omega, \mathcal{A}, \mathbb{P})$ and taking values in measurable spaces $(\mathcal{X}, \mathcal{B})$ and $(\mathcal{Y}, \mathcal{C})$, respectively, is the probability measure defined on $(\mathcal{X} \times \mathcal{Y}, \mathcal{B} \otimes \mathcal{C})$ by
$$
\mathbb{P}_{X, Y}(E) = \mathbb{P}((X, Y) \in E)
$$
for all $E \in \mathcal{B} \otimes \mathcal{C}$.
This is the same as the ordinary distribution of $(X, Y) : \Omega \to \mathcal{X} \times \mathcal{Y}$ when viewed as a single random variable defined on $\Omega$.
Also, $X$ and $Y$ are said to be independent if it holds that
$$
\mathbb{P}(X \in B, Y \in C) = \mathbb{P}(X \in B) \mathbb{P}(Y \in C)
$$
for all $B \in \mathcal{B}$ and $C \in \mathcal{C}$.
The independence condition can be rephrased in terms of the joint distribution of $X$ and $Y$: $X$ and $Y$ are independent if and only if
$$
\mathbb{P}_{X,Y}(B \times C) = \mathbb{P}_X(B) \mathbb{P}_Y(C)
$$
for all $B \in \mathcal{B}$ and $C \in \mathcal{C}$. That is, if and only if
$$
\mathbb{P}_{X, Y} = \mathbb{P}_X \otimes \mathbb{P}_Y.
$$
Thus, the joint distribution of $X$ and $Y$ is the product measure of the (marginal) distributions of $X$ and $Y$ precisely in the case that $X$ and $Y$ are independent.
If $X$ and $Y$ are dependent, then their joint distribution is not the product measure of the marginal distributions.
Computing Expectations over Joint Distributions
If $X$ and $Y$, as above, are independent and $f : \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$ is a measurable function (satisfying either non-negativity or integrability with respect to $\mathbb{P}_{X,Y}$), then Fubini's theorem allows you to compute
$$
\begin{aligned}
\mathbb{E}[f(X, Y)]
&= \int_\Omega f(X(\omega), Y(\omega)) \, \mathbb{P}(d\omega) &&
\text{(def. of expectation)}
\\
&= \int_{\mathcal{X} \times \mathcal{Y}} f(x, y) \, \mathbb{P}_{X, Y}(d(x, y)) &&
\text{(change of variables)}
\\
&= \int_{\mathcal{X} \times \mathcal{Y}} f(x, y) \, \mathbb{P}_X\otimes\mathbb{P}_Y(d(x, y)) &&\text{(independence)}
\\
&= \int_{\mathcal{Y}} \left(\int_{\mathcal{X}} f(x, y) \, \mathbb{P}_X(dx)\right) \, \mathbb{P}_Y(dy) &&\text{(Fubini's theorem)}
\end{aligned}
$$
However, if $X$ are $Y$ are not independent, then this argument won't work. Instead, if you want to break an expectation of $f(X,Y)$ into an integral over $\mathcal{X}$ followed by an integral over $\mathcal{Y}$, as we did above, you need to know something about the conditional distribution of $X$ given $Y$.
For what follows, suppose $(\mathcal{X},\mathcal{B})$ and $(\mathcal{Y}, \mathcal{C})$ are "sufficiently nice" measurable spaces, meaning that they admit conditional distributions (this will happen for most spaces in practice; a sufficient condition is being standard Borel).
Then if $\mathbb{P}_{X\mid Y} : \mathcal{B} \times \mathcal{Y} \to [0, 1]$ is a version of the conditional distribution of $X$ given $Y$, then we can proceed similarly to the calculations above:
$$
\begin{aligned}
\mathbb{E}[f(X, Y)]
&= \int_{\mathcal{X} \times \mathcal{Y}} f(x, y) \, \mathbb{P}_{X, Y}(d(x, y))
\\
&= \int_{\mathcal{Y}} \left(\int_{\mathcal{X}} f(x, y) \, \mathbb{P}_{X\mid Y}(dx, y)\right) \, \mathbb{P}_Y(dy) \\
&= \mathbb{E}[\mathbb{E}[f(X, Y) \mid Y]]
\end{aligned}
$$
(in fact, the formula $\mathbb{E}[f(X, Y)] = \mathbb{E}[\mathbb{E}[f(X, Y) \mid Y]]$ holds even without considering conditional distributions (proof), but it's arguably harder to compute in that case).
If $X$ and $Y$ are independent, then it happens that $\mathbb{P}_{X\mid Y}(B, y) = \mathbb{P}_X(B)$ for every $B \in \mathcal{B}$ and $\mathbb{P}_Y$-almost every $y \in Y$.
In this case, the calculation reduces to the first computation above.
In practice, the conditional distribution $\mathbb{P}_{X\mid Y}$ will usually be given by a conditional density $p_{X\mid Y} : \mathcal{X} \times \mathcal{Y} \to [0, \infty)$ of $X$ given $Y$ with respect to some dominating measure $\mu$ on $(\mathcal{X}, \mathcal{B})$, yielding
$$
E[f(X, Y)]
= \int_{\mathcal{Y}} \left(\int_{\mathcal{X}} f(x, y) p_{X \mid Y}(x, y) \, \mu(dx)\right) \, \mathbb{P}_Y(d y).
$$
