3
$\begingroup$

Why is the Empirical Probability Distribution Function based on the Cumulative Probability Distribution Function?

I have always seen the Empirical Distribution Function to have a "staircase" style shape, similar to a cumulative probability distribution function:

enter image description here

Is there any reason that the Empirical Distribution Function always appears in the shape of a "staircase" and estimates the Cumulative Probability Distribution Function instead of the Non-Cumulative Probability Distribution Function? Or is this a trivial matter - can the Empirical Probability Distribution be easily transformed into a Non-Cumulative Probability Distribution Function?

Can someone please explain this?

Thanks!

Note: Apparently F-hat (t) is an unbiased estimator of the true Cumulative Probability Distribution function F(t).

References:

$\endgroup$
7
  • 5
    $\begingroup$ As opposed to what, a density function or histogram? In some sense, a “distribution” is a CDF. $\endgroup$
    – Dave
    Nov 8 '21 at 4:33
  • 1
    $\begingroup$ @ Dave: Thank you for your reply! I am just confused - why doesn't the empirical distribution look like a "bell curve", and instead look like a "staircase"? Thank you! $\endgroup$
    – stats555
    Nov 8 '21 at 4:37
  • 2
    $\begingroup$ It’s just a matter of terminology. The bell curve is a density, not a distribution. (That sounds a little ridiculous, yes, but it’s technically correct.) If you plot the “distribution” of a Gaussian random variable, it will look like a smoothed staircase (perhaps like a slide or a skateboarding ramp). $\endgroup$
    – Dave
    Nov 8 '21 at 4:47
  • 3
    $\begingroup$ @Dave "The bell curve is a density, not a distribution." Part of me is all "eh." And another part of me wants to understand if there is something deep to appreciate about the probability density function not representing a distribution, but the integral of that PDF representing a distribution… surely they are just two "images" of one distribution… just as the histogram and eCDF plot both give insights? $\endgroup$
    – Alexis
    Nov 8 '21 at 4:55
  • $\begingroup$ There is a distinction: The density of the sum of two or more independent variables is the convolution of their densities. Thus the central limit theorem can be interpreted as a statement about the properties of density functions under convolution: the convolution of a number of density functions tends to the normal density as the number of density functions increases without bound. These theorems require stronger hypotheses than the forms of the central limit theorem that use (cumulative) distribution functions. Theorems of this type are often called local limit theorems. $\endgroup$ Nov 8 '21 at 14:45
16
$\begingroup$

The empirical distribution function $\hat{F}(\cdot)$

  1. is a step function by construction, since it puts a probability (Dirac) mass of $1/n$ on every term in the sample, $(x_1,\ldots,x_n)$, hence jumps by the same factor $1/n$ from one observation to the next. As a result, it is not everywhere differentiable and cannot be associated with a probability density function, meaning there is no density equivalent to the empirical distribution function (at least wrt a continuous measure like the Lebesgue measure).

  2. is "empirical" in the sense that it is based on an iid sample, $(x_1,\ldots,x_n)$, as opposed to the true cumulative distribution function, $F(\cdot)$, of the sample

  3. is a proper cumulative distribution function (cdf), corresponding to an average of Dirac distributions over the sample, $(x_1,\ldots,x_n)$

  4. estimates the true cdf $F$ with the property of being unbiased that$$\mathbb{E}^F[\hat{F}(x)]=F(x)$$for all $x$'s, and

  5. is converging a.s. to the true cdf $F$ for the uniform convergence norm. This is the Glivenko-Cantelli Theorem.

What is a non-cumulative distribution function?

$\endgroup$
10
$\begingroup$

The distribution function is a 'function to describe the distribution'.

But several functions can be used to describe a distribution, so the 'distribution function' may refer to different things. See for instance: Are the terms probability density function and probability distribution (or just "distribution") interchangeable?

Mostly used is the cumulative distribution function (CDF) because it uniquely defines a distribution, and it may be considered the density function. (I believe that the characteristic function and the cumulant generating function are also sometimes referred to with the term 'the distribution function').

According to this list of earliest uses of statistical terms the term 'distribution function' first occurred in 1919 in the German language literature (R. von Mises' "Grundlagen der Wahrscheinlichkeitsrechnung") and in 1935 in English language literature ( J. L. Doob's "The Limiting Distributions of Certain Statistics"). There is some works in English from 1933 by Aurel Witner, for instance "On the Stable Distribution Laws".

In those works by von Mises, Doob and Witner, the 'distribution function' is defined as what we know more commonly now as the cumulative distribution function. But there are around that time other uses of 'distribution function'. For instance in Nordic literature (more precise the Scandinavian Actuarial Journal) the term 'verteilungfunktion' occurs in 1919 as the probability density or frequency distribution, see Hongström and Hagström . We also see Wishart and Bartlett use the term 'distribution function' in 1933 referring to the probability density function "The generalised product moment distribution in a normal system" and Wilks in 1932.

Empirical distribution

So the 'empirical distribution' refers to an empirical estimate of the cumulative distribution function.

Below you see an example with a sample from a standard normal distribution.

CDF

Empirical frequency distribution

If the observations are discrete then instead of the probabilities $P(X \leq x)$ we could also describe the probabilities $P(X = x)$. This is also called the probability mass function(PMF).

Below is an example with the data from 'illustration I' in Pearson's article on the chi-squared statistic to test the goodness of fit for frequency curves.

The following data are due to Professor W.F.R. Weldon, F.R.S., and give the observed frequency of dice with 5 or 6 points when a cast of twelve dice was made 26 306 times:

PMF

Empirical density distribution

why doesn't the empirical distribution look like a "bell curve"

The bell curve is a probability density function (PDF). It is a density of the probability mass. The density function does not express probabilities, like the above $P(X \leq x)$ and $P(X = x)$. So we can not estimate the density function empirically by observing frequencies in a sample.

However, what sometimes is done is bin the data and create a histogram like the PMF case above. Other ways are estimating the density by some smoothening of the observed data.

Below is an example of estimating the normal distribution PDF with a kernel smoother. The sampled points are illustrated in the image as points at the top.

PDF

$\endgroup$
11
  • $\begingroup$ M. FRÉCHET AND J. SHOHAT 1931 "A proof of the generalized second-limit theorem in the theory of probability" call the CDF 'law of probability' $\endgroup$ Nov 8 '21 at 14:48
  • $\begingroup$ Great links! What is the meaning of the curve linking the points on the pmf of the Binomial? $\endgroup$
    – Xi'an
    Nov 9 '21 at 10:17
  • $\begingroup$ @Xi'an that curve might indeed be confusing. The points are the theoretical PMF. I added the curve to make the points more salient, but the function is not truly continuous. Should I delete it? Maybe we have a question about this "how to plot a pmf on top of a histogram?" or should create it? $\endgroup$ Nov 9 '21 at 11:25
  • $\begingroup$ I think this is confusing. which is why I made this tongue-in-cheek comment, as it gives the impression of a (continuous) density co-existing with the discrete nature of the distribution. An histogram is actually similarly dubious as a bar plot for the observed values would seem more appropriate. $\endgroup$
    – Xi'an
    Nov 9 '21 at 19:46
  • $\begingroup$ @Xi'an I agree, the continuous line has disadvantages and I will change it (I do somehow like to use lines in place of a pmf, because it makes the function more clear, but that is a personal preference). $\endgroup$ Nov 9 '21 at 20:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.