45
$\begingroup$

I read here that given a sample $ X_1,X_2,...,X_n $ from a continuous distribution with cdf $ F_X $, the sample corresponding to $ U_i = F_X(X_i) $ follows a standard uniform distribution.

I have verified this using qualitative simulations in Python, and I was easily able to verify the relationship.

import matplotlib.pyplot as plt
import scipy.stats

xs = scipy.stats.norm.rvs(5, 2, 10000)

fig, axes = plt.subplots(1, 2, figsize=(9, 3))
axes[0].hist(xs, bins=50)
axes[0].set_title("Samples")
axes[1].hist(
    scipy.stats.norm.cdf(xs, 5, 2),
    bins=50
)
axes[1].set_title("CDF(samples)")

plt.show()

Resulting in the following plot:

Plot showing the sample of a normal distribution and the cdf of the sample.

I am unable to grasp why this happens. I assume it has to do with the definition of the CDF and it's relationship to the PDF, but I am missing something ...

I would appreciate it if someone could point me to some reading on the subject or help me get some intuition on the subject.

EDIT: The CDF looks like this:

CDF of the sampled distribution

$\endgroup$
8
  • 4
    $\begingroup$ Compute the cdf of $F_X(X)$. $\endgroup$
    – Zhanxiong
    Commented Jul 15, 2015 at 16:50
  • 3
    $\begingroup$ You would find a proof of this property (for continuous rv's) in any book about simulation as this is the basis of the inverse cdf simulation method. $\endgroup$
    – Xi'an
    Commented Jul 15, 2015 at 17:06
  • 5
    $\begingroup$ Also try google-ing probability integral transform $\endgroup$ Commented Jul 15, 2015 at 17:13
  • 4
    $\begingroup$ @Xi'an It is good to point out the conclusion holds only for continuous random variables. Sometimes this result is mistakenly used for discrete random variables. On the other hand, also note many proofs involves the step $P(F(X) \leq x) = P(X \leq F^{-1}(x))$ in which assumes the strict monotonicity of $F$, which is also a too strong assumption. The following link provides a rigorous summary on this topic:people.math.ethz.ch/~embrecht/ftp/generalized_inverse.pdf $\endgroup$
    – Zhanxiong
    Commented Jul 15, 2015 at 17:20
  • $\begingroup$ @Zhanxiong the only condition necessary for $F$ is that it is càdlàg. $\endgroup$
    – AdamO
    Commented Jul 15, 2015 at 17:26

5 Answers 5

45
$\begingroup$

Assume $F_X$ is continuous and increasing. Define $Z = F_X(X)$ and note that $Z$ takes values in $[0, 1]$. Then $$F_Z(x) = P(F_X(X) \leq x) = P(X \leq F_X^{-1}(x)) = F_X(F_X^{-1}(x)) = x.$$ The derivative of $\frac{d}{dz}F_Z(x) = f_Z(x) = 1$ so $Z$ is uniformly distributed $[0, 1]$.

Another way to see this: A uniform random variable $U$ taking values in $[0, 1]$ has $F_U(x) = \int_R f_U(u)\,du =\int_0^x \,du =x$. So $F_Z(x) = F_U(x)$ for every $x\in[0, 1]$. Since $Z$ and $U$ have the same distribution function $Z$ must also be uniform on $[0, 1]$.

$\endgroup$
4
  • $\begingroup$ Does it follow that Z has a uniform(0, 1) distribution? $\endgroup$ Commented Nov 13, 2019 at 3:29
  • $\begingroup$ @StatsSorceress Yes, you are right. $Z$ has a standard uniformly distribution on $(0,1).$ $\endgroup$
    – Idonknow
    Commented Dec 28, 2019 at 12:04
  • $\begingroup$ Why is $Z$ a random variable? If the CDF is a random variable, the pdf must be also a random variable but according to this question it seems that isn't the case. $\endgroup$
    – ado sar
    Commented Dec 19, 2023 at 18:17
  • $\begingroup$ @adosar: $Z$ is a random variable as a transform of another random variable $X$. $\endgroup$
    – Xi'an
    Commented Feb 11 at 15:49
17
$\begingroup$

Intuitively, perhaps it makes sense to think of $F(x)$ as a percentile function, e.g. $F(x)$ of a randomly generated sample from the DF $F$ is expected to fall below $x$. Alternately $F^{-1}$ (think inverse images, not a proper inverse function per se) is a "quantile" function. That is, $x = F^{-1}(p)$ is the point $x$ behind which falls $p$ proportion of the sample. The functional composition is measurably commutative $F \circ F^{-1} =_\lambda F^{-1} \circ F$.

The uniform distribution is the only distribution having a quantile function equal to a percentile function: they are the identity function. So the image space is the same as the probability space. $F$ maps continuous random variables into a (0, 1) space with equal measure. Since for any two percentiles, $a < b$, we have $P(F^{-1}(a) < x < F^{-1}(b)) = P(a < F(X) < b) = b-a$

$\endgroup$
1
  • $\begingroup$ I struggled for hours, but finally it clicked why the derived random variable $Y = F(X)$ is uniformly distributed. Your answer really helped, thanks a lot. It seems very much like in algebra where 1 was the multiplicative identity. $\endgroup$
    – Aditya P
    Commented Sep 21, 2018 at 12:50
13
$\begingroup$

Here's some intuition. Let's use a discrete example.

Say after an exam the students' scores are $X = [10, 50, 60, 90]$. But you want the scores to be more even or uniform. $h(X) = [25, 50, 75, 100]$ looks better.

One way to achieve this is to find the percentiles of each student's score. Score $10$ is $25\%$, score $50$ is $50\%$, and so on. Note that the percentile is just the CDF. So the CDF of a sample is "uniform".

When $X$ is a random variable, the percentile of $X$ is "uniform" (e.g. the number of $X$'s in $0-25$ percentile should be the same as the number of $X$'s in $25-50$ percentile). Therefore the CDF of $X$ is uniformly distributed.

$\endgroup$
6
$\begingroup$

As I commented under several posts above, the key of a rigorous (and succinct) proof to the general continuous $F$ (that is, $F$ is not necessarily strictly increasing) is by introducing the quantile function: \begin{align*} \varphi(u) = \inf\{x: F(x) \geq u\}, \quad 0 < u < 1. \tag{1}\label{1} \end{align*} Note that this function is well defined for any distribution function $F$, regardless it has discontinuities or strictly increasing. This can also be used as a definition of the population $u$-quantile of a random variable whose distribution function is $F$. A graph of $\varphi(\cdot)$, which stresses its values at a discontinuity and a plateau point, is shown as follows (source: Probability and Measure (3rd edition), p.189):

Quantile Function

The most important property of $\varphi$ is its relationship with $F$: for any $0 < u < 1$: \begin{align*} F(\varphi(u)-) \leq u \leq F(\varphi(u)), \tag{2}\label{2} \end{align*} where $F(\varphi(u)-) = \lim\limits_{x \uparrow \varphi(u)}F(x)$. From the graph, inequality $\eqref{2}$ is evident. The proof of $\eqref{2}$ will be relegated to the end of this answer. Note that for a general $F$, $\eqref{2}$ is not equivalent to $F(\varphi(u)) = u$, as many "proofs" falsely claimed or relied on.

However, under the condition that $F$ is continuous (everywhere), $\eqref{2}$ of course reduces to $F(\varphi(u)) \leq u \leq F(\varphi(u))$, whence $F(\varphi(u)) = F(\varphi(u)-) = u$. It then follows that \begin{align*} P[F(X) < u] &= 1 - P[F(X) \geq u] \\ &= 1 - P[X \geq \varphi(u)] = P[X < \varphi(u)] = F(\varphi(u)-) = u. \tag{3}\label{3} \end{align*} In the second equality above, we used the property that $\{F(X) \geq u\} = \{X \geq \varphi(u)\}$, whose proof is also placed at the end of the answer. Now by $\eqref{3}$ and the continuity (from above) of the probability measure, we have for $0 < u < 1$: \begin{align*} P[F(X) \leq u] = \lim\limits_{h \downarrow 0}P[F(X) < u + h] = \lim\limits_{h \downarrow 0}(u + h) = u. \end{align*} This shows that $F(X) \sim U(0, 1)$, and the proof is complete.


Proof of $\eqref{2}$

By definition $\eqref{1}$, if $y < \varphi(u)$, then $F(y) < u$ (otherwise, $\varphi(u)$ would not be a lower bound of the set $S_u := \{x: F(x) \geq u\}$ as $F(y) \geq u$ and $y < \varphi(u)$). This implies that $F(\varphi(u)-) = \lim\limits_{y \uparrow \varphi(u)}F(y) \leq u$.

On the other hand, for every $n \in \mathbb{N}$, since $\varphi(u) + n^{-1}$ is not the infimum of $S_u$, there exists $x_n \in S_u$ such that $x_n < \varphi(u) + n^{-1}$, whence $F(\varphi(u) + n^{-1}) \geq F(x_n) \geq u$ by the monotonicity of $F$. This implies by the right-continuity of $F$ that $F(\varphi(u)) = \lim\limits_{n \to \infty}F(\varphi(u) + n^{-1}) \geq u$.

This completes the proof of $\eqref{2}$.

Proof of $\{F(X) \geq u\} = \{X \geq \varphi(u)\}$

If $F(X) \geq u$, then $X \in \{x: F(x) \geq u\}$, whence $X \geq \inf\{x: F(x) \geq u\} = \varphi(u)$. This shows that $\{F(X) \geq u\} \subseteq \{X \geq \varphi(u)\}$.

On the other hand, if $X \geq \varphi(u)$, then $F(X) \geq F(\varphi(u))$ by the monotonicity of $F$. As we have proven $F(\varphi(u)) \geq u$, it follows that $F(X) \geq u$, hence $\{X \geq \varphi(u)\} \subseteq \{F(X) \geq u\}$.

$\endgroup$
1
  • 1
    $\begingroup$ Same approach has been used in introducing the cdf by Sidney Resnick (along with the same diagram!) in his book (although the approach itself is not unique and ingenious in itself). Anyway +1. $\endgroup$ Commented Nov 24, 2023 at 3:46
0
$\begingroup$

Yes, CDF of a random variable has uniform distribution in the interval $(0, 1)$ because the distribution of its CDF has a form $F(Y \leq y) = y.$ In a formal way, if $X $ is a random variable, then $ F(X) \sim \textrm{Unif}(0,1). $

Proof:

To understand this special property, the key is to understand the inverse function which is generated from the relationship between $ X $ and its CDF. For example, if $ X$ is a Pareto distribution with parameter a and has CDF as:

$$ F(x)= \begin{cases} 1 - 1/ x^a, &\textrm{if}~ x > 1,\\ 0 &\textrm{if}~ x \leq 1\end{cases}$$

The inverse function is $ y = F(x) = 1 - 1/x^a.$

Thus we can know $x$ if we know $y$ and vice versa, leading us to the inverse function:

$$ x = 1/(1-y)^{1/a}$$

in a formal way: $F^{-1}$ is the inverse function of $F.$ If $F(x) = y $ then $F^{-1}(y) = x.$ In the above example:

$$ F^{-1}(y) = 1/(1-y)^{1/a} $$

For an easier example, if we have half of people score below 70 then its CDF $F(70) = 1/2,$ its inverse function $F^{-1}(1/2) = 70.$ Now we can see the act of inversing a function is to change from x to y and from $y$ to $x.$ Thus $F^{-1}(F(x)) = x$ and $F^{-1}(F(y)) = y.$

Now we are able to generate the proof we need. X is random variable. We have $F(x) = y $then:

$F^{-1}(y) = x,$ and $F(F^{-1}(y)) = y $ because it is just another way to say $F(x) = y . $

We find the distribution of variable Y by finding its cumulative distribution:

$$\Pr(Y < y) = \Pr(F(X) < y) = \Pr(X < F^{-1}(y)) = F(F^{-1}(y)) = y;$$

therefore $\Pr(Y < y) = y $ which makes it a uniform distribution $\textrm{Unif}(0,1). $

$\endgroup$
2
  • 3
    $\begingroup$ I have done for you this time. And I am pretty sure there are typographical errors which I have not fixed. Check yourself and correct them where needed. $\endgroup$ Commented Oct 16, 2023 at 4:41
  • 1
    $\begingroup$ @User1865345 thank you for your kindness. I will check the link you provided. Sorry, this is the first time I join this website. Have a nice day! $\endgroup$ Commented Oct 16, 2023 at 14:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.