6
$\begingroup$

I'm attempting to understand the proof of the probability integral transform in [1].

First they define $Y = F_X ( X )$. Yet, the cumulative distribution function is defined in [2] as $$F_X ( x ) = P(X\leq x).$$

Q1. Can you define $F_X ( X )$ (preferred answer would provide a measure-theoretic definition [4], but any definitions are welcome)?

Continuing to read from [1], there is a step such that

$$P(F_X ( X ) \leq y) = P( X \leq F^{-1}_X ( y )).$$

Q.2. Can you explain how come this equality holds?

I imagine its related to the properties shown in [3]. Namely that $$F^{-1}(F(x)) \leq x$$ and $$F(F^{-1}(p)) \geq p.$$ Even so, such seems to me to be insufficient to understand why the equality holds.

[1] https://en.wikipedia.org/wiki/Probability_integral_transform#Proof

[2] https://en.wikipedia.org/wiki/Cumulative_distribution_function

[3] https://en.wikipedia.org/wiki/Cumulative_distribution_function#Inverse_distribution_function_(quantile_function)

[4] https://en.wikipedia.org/wiki/Random_variable#Measure-theoretic_definition

$\endgroup$
7
  • $\begingroup$ What is wrong with the definition of cumulative distribution function in your eyes, and why does it puzzle you that one can define $Y = F_X(x) = P(X \leq x)$? $\endgroup$
    – jbowman
    Aug 29, 2019 at 14:59
  • $\begingroup$ Nothing is wrong with the definition of the cumulative distribution function. Its does not puzzle me that $y = F_X(x) = P(X\leq x)$. I am puzzled about the random variable $Y$ defined as $Y = F_X(X)$ . $\endgroup$ Aug 29, 2019 at 15:01
  • 2
    $\begingroup$ If $X$ is random, then $F_X(X)$ is random, so $Y = F_X(X)$ is random. $\endgroup$
    – jbowman
    Aug 29, 2019 at 15:01
  • 1
    $\begingroup$ Note also that by the definition of the inverse of a function, $F^{-1}(F(x)) = x$, not just $\leq x$. $\endgroup$
    – jbowman
    Aug 29, 2019 at 15:03
  • 2
    $\begingroup$ In this case, we use the generalized inverse distribution function; see the answer to stats.stackexchange.com/questions/212813/… for more detail. $\endgroup$
    – jbowman
    Aug 30, 2019 at 1:37

2 Answers 2

7
$\begingroup$

The notation is getting in the way, so let's simplify it.

Let $X:\Omega\to\mathbb{R}$ be a random variable with a distribution function $F_X$ defined by

$$F_X(x) = \Pr(X \le x) = \Pr(\{\omega\in\Omega\mid X(\omega)\le x\})$$

for all real numbers $x.$ The axioms of probability imply $F_X$ is non-decreasing and at any point of discontinuity its value is the limit from the right of its values (from left to right, its graph jumps up to its value rather than up from its value).

Consider any measurable function $h:\mathbb{R}\to\mathbb{R}$ with these properties (whether or not it actually is a probability function), as graphed here:

Figure

Because $h$ is measurable, the composition $Y = h \circ X:\Omega\to\mathbb R$ is also a random variable. When $X$ has the value $x,$ $Y$ has the value $h(x):$ you can read it directly off the graph.

We will want to go backwards from values of $Y$ to corresponding values of $X$ by inverting $h.$ Two possible behaviors make this problematic, as shown by the dotted colored lines in the figure.

  1. Where $h$ has a jump from a value $a$ to a value $b$ at an argument $x,$ define the inverse of $h$ (written $h^{-1}$) at any point in the interval $[a,b)$ to be the limiting height of all points strictly to the left of $x.$ For instance, for any $q_1$ with $a \le q_1 \lt b$ in the figure, the values of $h^{-1}(q_1)$ are all the same, equal to the height of the open circle (the "base" of the jump).

  2. Wherever $h$ is horizontal at a height of $q_2,$ there is an entire closed interval $[a,b]$ of values for which $h(x) = q_2$ whenever $a \le q_2 \le b.$ Define $h^{-1}(q_2)$ to be the largest such value (or infinity if there is no largest value).

These definitions imply

$$h(h^{-1}(y))=y\tag{*}$$

whenever $y$ is in the image of $h$ and otherwise $h(h^{-1}(y)) \ge y.$ The definitions are arranged so that--as the figure clearly shows--whenever $y$ is a possible value of $Y,$

$$\Pr(Y\le y) = \Pr(h(X)\le y) = \Pr(X \le h^{-1}(y)) = F_X(h^{-1}(y))\tag{**}$$

and otherwise (where $y$ is in the middle of a jump),

$$\Pr(Y\le y) = \Pr(h(X)\le y) = \Pr(X \lt h^{-1}(y)).$$

In particular, the mere substitution of $F_X$ for $h$ (whose values lie in the interval $[0,1]$) in $(*)$ and $(**)$ shows that for any $p$ in the image of $F_X,$

$$\Pr(Y \le p) = \Pr(F_X(X)\le p) = F_X(F_X^{-1}(p)) = p.$$

(I hope this makes it clear that the subscript "$X$" on $F$ is not acting as a random variable in these expressions, which perhaps is the most confusing aspect of the notation; $F_X$ is a completely determinate, non-random function.)

When $F_X$ is everywhere continuous (that is, $X$ is a continuous random variable), this is true for all $p\in [0,1]$. The equation $\Pr(Y\le p) = p$ for $0\le p \le 1$ defines the uniform distribution on $[0,1].$ We have concluded:

Transforming the continuous random variable $X$ via its probability function $F_X$ creates a random variable $Y=F_X(X)$ that has the uniform distribution on the interval $[0,1].$

This is the probability integral transform, or PIT. Although no integration was needed to define it, notice that absolutely continuous random variables $X$ have densities $f_X$ with $f_X(x)\mathrm{d}x = \mathrm{d}F_X(x),$ whence substituting $y = F_X(x)$ in the integral for the expectation of any measurable function $g$ gives

$$E_X[g(X)] = \int_{\mathbb R} g(x) f_X(x) \mathrm{d}x = \int_{\mathbb R} g\left(F_X^{-1}(y)\right) \mathrm{d} y = E_Y\left[g\circ F_X^{-1}(Y)\right].$$

In other words, the PIT converts integration with respect to the density $f_X(x)\mathrm{d}x$ into integration with respect to $\mathrm{d}y.$

$\endgroup$
0
3
$\begingroup$

After having done some homework on the subject, I think I've got a better handle on the proof that I find in [1]. I wanted to take an opportunity to set down my understanding for pedagogic purposes.

Scope: I'm going to limit this answer to the case of a strictly monotonic cumulative distribution function. Its my understanding that, in his answer to this post, @whuber considers a more general situation. In addition, this is not a formal proof, just the outline of my understanding. So certain details are likely omitted.

Rudimentary derivation:

By $X$ I denote a real-valued random variable. By $x$ I denote a real variable. By $F_X(x)$ I denote the cumulative marginal density function of the random variable $X$, where $$F_X(x) = P(X\leq x).\quad \textrm{Eq. 1}$$

By $Y$ I denote a new random variable defined in terms of $X$ as $$Y = F_X(X). \quad \textrm{Eq. 2}$$

By $y$ I denote a real number in the interval $[0,1]$. By $F_Y(y)$ I denote the cumulative marginal density function of the random variable $Y$, where $$F_Y(y) = P(Y\leq y).$$

From Eq. 2, I can subtitute $F_X(X)$ in place of $Y$. I find $$F_Y(y) = P(F_X(X) \leq y).$$

Since $F_X$ is assumed to be strictly increasing, it is invertible. When I apply the inverse $F^{-1}_X$ to both sides of the inequality in the argument I find $F^{-1}_X(F_X(X)) \leq F^{-1}_X(y)$. Again, since $F_X$ is invertible, therefore $F^{-1}_X(F_X(X)) = X$. I continue with my central train of thought and write $$F_Y(y) = P( X \leq F^{-1}_X(y)). \quad \textrm{Eq. 3}$$

Next, by comparing Eq. 3 with Eq. 1, I find that

$$F_Y(y) = F_X(F_X^{-1}(y)). $$

Once again, since $F_X$ is invertible $$F_Y(y) = y. \quad \textrm{Eq. 4} $$

As I write in the scope, there are some details missing. Nonetheless, if one compares the result in Eq. 4 with the cumulative distribution function (CDF) given in the first table in [5], which reads $$\text{CDF} : \begin{cases} 0,&\text{for}~y<a, \\ \frac{y-a}{b-a},& \text{for}~y\in[a,b],~\text{and} \\ 1,& \text{for}~y>b ; \end{cases} $$ then one may see that Eq. 4 describes the cumulative distribution function of random variable with a standard-uniform distribution (i.e., $a=0$ and $b=1$).

Thus, the random variable $Y$, which is given by $Y = F_X(X)$, has a standard-uniform distribution.

Bibliography

[5] https://en.wikipedia.org/wiki/Uniform_distribution_(continuous)

$\endgroup$
1
  • 1
    $\begingroup$ This kind of analysis is useful for developing an understanding of the principles. Note, however, that few CDFs satisfy your criteria: they describe continuous distributions supported on the entire real line. This eliminates common distributions like the uniform, beta, exponential, Gamma, Pareto, etc. That's why we can't avoid the more general considerations needed to handle distribution functions that aren't everywhere strictly monotonic. $\endgroup$
    – whuber
    Nov 1, 2021 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.