What is the cross product of two probability distributions $P \times Q$?

Question

In many papers on machine learning and statistics, I encounter the following notation

Let $P$ be a distribution, and let $Q$ be another distribution.

Then the author creates an object $P \times Q$

However, this object is not defined.

It is not known if $\times$ means the usual product, cross product, cartesian product, or some other product.

Example: https://arxiv.org/pdf/1903.08560.pdf (Equation 1)

Can someone please offer an explanation to the precise mathematical operation $\times$ is doing to these distributions and offer at least a simple example when $P, Q$ are simple probability distributions such as the Gaussian probability density function?

In other words, can someone please provide simple examples of $F$ where $F = P \times Q$?

Answer 1

I haven't read the specifics of the paper in question but the author meant by $P_x\times P_\epsilon$ the product measure. This measure is used when you have to equip a measure to a new measurable space \begin{equation} \big(X_1\times X_2\times \cdots \times X_n, \sigma(\boldsymbol{\mathfrak A}_1\times\boldsymbol{\mathfrak A}_2\times \cdots \boldsymbol{\mathfrak A}_n)\big) \end{equation} created from the product of $n\geq 2$ measure spaces \begin{equation} \langle (X_i, \boldsymbol{\mathfrak A}_i, \mu_i)\rangle_{i=1}^n \end{equation}

defined as

\begin{align} \mu(A) :=&\mu_1(A_1) \cdot \mu_2(A_2) \cdot \cdots \cdot\mu_n(A_n) \\ ~A=&A_1\times A_2\times \cdots\times A_n\in\boldsymbol{\mathfrak A}_1\times \boldsymbol{\mathfrak A}_2\times\cdots \times \boldsymbol{\mathfrak A}_n \text{.} \end{align}

Same concept is applicable for probability measures.

For details, please check any measure theory book/probability book. For a quick reference, check the Wikipedia article.

Answer 2

In other words, can someone please provide simple examples of $F$ where $F=P\times Q$?

User1865345's answer is correct, however you need to know a bit of measure theory notation to understand it. Therefore I will add another one and try to explain in simple terms.

The notation $P\times Q$ stands for the joint distribution of two independent random variables, one of which has distribution denoted by $P$, and the other has a distribution denoted by $Q$.
In the paper you cited, they look at random variables $x_i$ and $\epsilon_i$ with distributions given by $P_x$ and $P_\epsilon$. The statement $$ (x_i,\epsilon_i)\sim P_x\times P_\epsilon $$ just means that the simultaneous probability for $x_i\in A$ and $\epsilon_i \in B$ is obtained as the product of the probabilities of $x_i\in A$ and of $\epsilon_i\in B$. This is a way of saying that the $x_i$ and the $\epsilon_i$ are independent.

The joint probability density of independent random variables is also obtained as the product of the two probability densities.

Say, we have a random variable $X\sim N(0,1)$ and another random variable $Y\sim N(0,2)$. If these are independent, their probability density is $$ \begin{align} f_{X,Y}(x,y) = f_X(x)\cdot f_Y(y) &= \frac{1}{\sqrt{2\pi}}\exp(-\frac{x^2}{2})\frac{1}{\sqrt{2\pi}\sqrt{2}}\exp(-\frac{y^2}{2\cdot 2}) \\&= \frac{1}{2\sqrt{2}\pi}\exp(-\frac{2x^2 +y^2}{4}). \end{align} $$ Now consider another random variable $\tilde{Y}=X+Z$, where $Z\sim N(0,1)$ is independent of $X$. Again, $\tilde{Y}\sim N(0,2)$, but $\tilde Y$ is no longer independent of $X$. The joint density is calculated using the formula for convolution of distributions, $$ \begin{align} f_{X,\tilde Y}(x,y) = f_X(x)f_Z(y-x) &= \frac{1}{\sqrt{2\pi}}\exp(-\frac{x^2}{2})\frac{1}{\sqrt{2\pi}}\exp(-\frac{(x-y)^2}{2}) \\&= \frac{1}{2\pi}\exp(-\frac{2x^2 +y^2-2xy}{2}). \end{align} $$

This extends to the case of more than two independent random variables in the same way: the joint (simultaneous) probability is simply the product of the corresponding probabilities for each of the variables. That is why the joint probability measure is called product measure. When the variables are not independent, the joint probability is not the product measure.

Answer 3

Expanding upon @User1865345's and @Ute's answers, here is another way of understanding product measures on $\mathcal B(\mathbb R^d)$, the Borel $\sigma$-algebra on $\mathbb R^d$, via their associated (multivariate) CDFs:

Recall that for each (multivariate) CDF $F: \mathbb R^d \to \mathbb R$ there is exactly one probability measure $\mathbb P_F$ on $\mathcal B(\mathbb R^d)$ with CDF $F$, i.e., $$ \mathbb P_F((-\infty, t_1] \times \ldots \times (-\infty, t_d]) = F(t_1, \ldots, t_d) \;\forall\,t_1, \ldots, t_d \in \mathbb R $$ holds for exactly one $\mathbb P_F$.

The CDF of the product measure $\mathbb P_{F_1} \times \cdots \times \mathbb P_{F_d}$ is, $\forall\,t_1, \ldots, t_d \in \mathbb R$, given by \begin{align} F_{\mathbb P_{F_1} \times \cdots \times \mathbb P_{F_d}}(t_1, \ldots t_d) &= (\mathbb P_{F_1} \times \cdots \times \mathbb P_{F_d})((-\infty, t_1] \times \ldots \times (-\infty, t_d]) \\ &= \mathbb P_{F_1}((-\infty, t_1]) \cdot \ldots \cdot \mathbb P_{F_1}((-\infty, t_d]) \\ &= F_1(t_1) \cdot \ldots \cdot F_d(t_d), \end{align} i.e., by the product of the corresponding marginal (univariate) CDFs.

Now, letting $X_1 \sim F_1, \ldots, X_d \sim F_d$, we find that $(X_1, \ldots, X_d) \sim \mathbb P_{F_1} \times \cdots \times \mathbb P_{F_d}$ iff the joint CDF of $(X_1, \ldots, X_d)$ factorizes into the marginal CDFs (i.e., iff $X_1, \ldots, X_d$ are independent).

In the example in the arXiv paper you link to, $P_x$ is a probability measure on $\mathcal B(\mathbb R^p)$ and $P_\epsilon$ is a probability measure on $\mathcal B(\mathbb R)$.
Thus, $(x_i, \epsilon_i) \sim P_x \times P_\epsilon$ in equation $(1)$ just means that the joint CDF $F_{x_i, \epsilon_i} \equiv F_{P_x \times P_\epsilon}$ of $(x_i, \epsilon_i)$ is the product of the marginal CDFs $F_{x_i} \equiv F_{P_x}$ and $F_{\epsilon_i} \equiv F_{P_\epsilon}$ of $x_i$ and $\epsilon_i$, respectively: $$ F_{x_i, \epsilon_i}(t_1, \ldots, t_p, t_{p+1}) = F_{x_i}(t_1, \ldots, t_p) \cdot F_{\epsilon_i}(t_{p+1}) \;\forall\,t_1, \ldots, t_{p+1} \in \mathbb R. $$