7
$\begingroup$

In many papers on machine learning and statistics, I encounter the following notation

Let $P$ be a distribution, and let $Q$ be another distribution.

Then the author creates an object $P \times Q$

However, this object is not defined.

It is not known if $\times$ means the usual product, cross product, cartesian product, or some other product.

Example: https://arxiv.org/pdf/1903.08560.pdf (Equation 1)

Can someone please offer an explanation to the precise mathematical operation $\times$ is doing to these distributions and offer at least a simple example when $P, Q$ are simple probability distributions such as the Gaussian probability density function?

In other words, can someone please provide simple examples of $F$ where $F = P \times Q$?

$\endgroup$
3
  • 4
    $\begingroup$ Isn't it simply the product measure? $\endgroup$ Commented Sep 10, 2023 at 3:57
  • $\begingroup$ @User1865345 Hi, could you please provide a reference to the "product measure" and make it into an answer? I will accept it. $\endgroup$
    – Fraïssé
    Commented Sep 10, 2023 at 6:01
  • $\begingroup$ It's the product measure, the underlying variables are assumed independent (in this example the independent variables and the additive error) $\endgroup$
    – seanv507
    Commented Sep 10, 2023 at 6:01

3 Answers 3

8
+25
$\begingroup$

In other words, can someone please provide simple examples of $F$ where $F=P\times Q$?

User1865345's answer is correct, however you need to know a bit of measure theory notation to understand it. Therefore I will add another one and try to explain in simple terms.

The notation $P\times Q$ stands for the joint distribution of two independent random variables, one of which has distribution denoted by $P$, and the other has a distribution denoted by $Q$.
In the paper you cited, they look at random variables $x_i$ and $\epsilon_i$ with distributions given by $P_x$ and $P_\epsilon$. The statement $$ (x_i,\epsilon_i)\sim P_x\times P_\epsilon $$ just means that the simultaneous probability for $x_i\in A$ and $\epsilon_i \in B$ is obtained as the product of the probabilities of $x_i\in A$ and of $\epsilon_i\in B$. This is a way of saying that the $x_i$ and the $\epsilon_i$ are independent.

The joint probability density of independent random variables is also obtained as the product of the two probability densities.

Say, we have a random variable $X\sim N(0,1)$ and another random variable $Y\sim N(0,2)$. If these are independent, their probability density is $$ \begin{align} f_{X,Y}(x,y) = f_X(x)\cdot f_Y(y) &= \frac{1}{\sqrt{2\pi}}\exp(-\frac{x^2}{2})\frac{1}{\sqrt{2\pi}\sqrt{2}}\exp(-\frac{y^2}{2\cdot 2}) \\&= \frac{1}{2\sqrt{2}\pi}\exp(-\frac{2x^2 +y^2}{4}). \end{align} $$ Now consider another random variable $\tilde{Y}=X+Z$, where $Z\sim N(0,1)$ is independent of $X$. Again, $\tilde{Y}\sim N(0,2)$, but $\tilde Y$ is no longer independent of $X$. The joint density is calculated using the formula for convolution of distributions, $$ \begin{align} f_{X,\tilde Y}(x,y) = f_X(x)f_Z(y-x) &= \frac{1}{\sqrt{2\pi}}\exp(-\frac{x^2}{2})\frac{1}{\sqrt{2\pi}}\exp(-\frac{(x-y)^2}{2}) \\&= \frac{1}{2\pi}\exp(-\frac{2x^2 +y^2-2xy}{2}). \end{align} $$

This extends to the case of more than two independent random variables in the same way: the joint (simultaneous) probability is simply the product of the corresponding probabilities for each of the variables. That is why the joint probability measure is called product measure. When the variables are not independent, the joint probability is not the product measure.

$\endgroup$
1
  • $\begingroup$ @Fraïssé, sorry I overlooked that you asked for an example of Gaussian random variables - I added one :-) $\endgroup$
    – Ute
    Commented Sep 18, 2023 at 18:16
12
$\begingroup$

I haven't read the specifics of the paper in question but the author meant by $P_x\times P_\epsilon$ the product measure. This measure is used when you have to equip a measure to a new measurable space \begin{equation} \big(X_1\times X_2\times \cdots \times X_n, \sigma(\boldsymbol{\mathfrak A}_1\times\boldsymbol{\mathfrak A}_2\times \cdots \boldsymbol{\mathfrak A}_n)\big) \end{equation} created from the product of $n\geq 2$ measure spaces \begin{equation} \langle (X_i, \boldsymbol{\mathfrak A}_i, \mu_i)\rangle_{i=1}^n \end{equation}

defined as

\begin{align} \mu(A) :=&\mu_1(A_1) \cdot \mu_2(A_2) \cdot \cdots \cdot\mu_n(A_n) \\ ~A=&A_1\times A_2\times \cdots\times A_n\in\boldsymbol{\mathfrak A}_1\times \boldsymbol{\mathfrak A}_2\times\cdots \times \boldsymbol{\mathfrak A}_n \text{.} \end{align}

Same concept is applicable for probability measures.

For details, please check any measure theory book/probability book. For a quick reference, check the Wikipedia article.

$\endgroup$
6
  • $\begingroup$ Is it possible to provide a concrete example of product measure function? For example, suppose $P_x$ and $P_\epsilon$ are two Gaussian probability density functions (as the authors intended), then what is the product measure function resulting from the operation $P_x \times P_\epsilon$? $\endgroup$
    – Fraïssé
    Commented Sep 10, 2023 at 9:03
  • $\begingroup$ Product measure $\mathbb P_{X_1X_2} = \mathbb P_{X_1}\times\mathbb P_{X_2}$ when $X_i\overset{\textrm{iid}}{\sim}\mathcal N(0, \sigma^2) $ and the pdf would be $f_1\times f_2.$ $\endgroup$ Commented Sep 10, 2023 at 15:13
  • $\begingroup$ @User1865345 : Can one think of the pdf of the product measure in the $n=2$ case to be the joint density or does the product measure always assume independence of $x$ and $y$ so that the joint density is the product of the two pdf's ? thanks. $\endgroup$
    – mlofton
    Commented Sep 11, 2023 at 3:17
  • $\begingroup$ The latter is the correct observation @mlofton. $\endgroup$ Commented Sep 11, 2023 at 3:24
  • $\begingroup$ @User1865345 I appreciate the clarification. $\endgroup$
    – mlofton
    Commented Sep 11, 2023 at 14:55
5
$\begingroup$

Expanding upon @User1865345's and @Ute's answers, here is another way of understanding product measures on $\mathcal B(\mathbb R^d)$, the Borel $\sigma$-algebra on $\mathbb R^d$, via their associated (multivariate) CDFs:

Recall that for each (multivariate) CDF $F: \mathbb R^d \to \mathbb R$ there is exactly one probability measure $\mathbb P_F$ on $\mathcal B(\mathbb R^d)$ with CDF $F$, i.e., $$ \mathbb P_F((-\infty, t_1] \times \ldots \times (-\infty, t_d]) = F(t_1, \ldots, t_d) \;\forall\,t_1, \ldots, t_d \in \mathbb R $$ holds for exactly one $\mathbb P_F$.

The CDF of the product measure $\mathbb P_{F_1} \times \cdots \times \mathbb P_{F_d}$ is, $\forall\,t_1, \ldots, t_d \in \mathbb R$, given by \begin{align} F_{\mathbb P_{F_1} \times \cdots \times \mathbb P_{F_d}}(t_1, \ldots t_d) &= (\mathbb P_{F_1} \times \cdots \times \mathbb P_{F_d})((-\infty, t_1] \times \ldots \times (-\infty, t_d]) \\ &= \mathbb P_{F_1}((-\infty, t_1]) \cdot \ldots \cdot \mathbb P_{F_d}((-\infty, t_d]) \\ &= F_1(t_1) \cdot \ldots \cdot F_d(t_d), \end{align} i.e., by the product of the corresponding marginal (univariate) CDFs.

Now, letting $X_1 \sim F_1, \ldots, X_d \sim F_d$, we find that $(X_1, \ldots, X_d) \sim \mathbb P_{F_1} \times \cdots \times \mathbb P_{F_d}$ iff the joint CDF of $(X_1, \ldots, X_d)$ factorizes into the marginal CDFs (i.e., iff $X_1, \ldots, X_d$ are independent).

In the example in the arXiv paper you link to, $P_x$ is a probability measure on $\mathcal B(\mathbb R^p)$ and $P_\epsilon$ is a probability measure on $\mathcal B(\mathbb R)$.
Thus, $(x_i, \epsilon_i) \sim P_x \times P_\epsilon$ in equation $(1)$ just means that the joint CDF $F_{x_i, \epsilon_i} \equiv F_{P_x \times P_\epsilon}$ of $(x_i, \epsilon_i)$ is the product of the marginal CDFs $F_{x_i} \equiv F_{P_x}$ and $F_{\epsilon_i} \equiv F_{P_\epsilon}$ of $x_i$ and $\epsilon_i$, respectively: $$ F_{x_i, \epsilon_i}(t_1, \ldots, t_p, t_{p+1}) = F_{x_i}(t_1, \ldots, t_p) \cdot F_{\epsilon_i}(t_{p+1}) \;\forall\,t_1, \ldots, t_{p+1} \in \mathbb R. $$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.