# Small sample size fails to approach inverse CDF

When sample size $$n$$ gets large, we know that a sorted set of the $$n$$ samples approaches the inverse cumulative distribution function (CDF) sampled at $$\frac{1}{n}, \frac{2}{n}, \dots, \frac{n}{n}$$.

But when we reduce $$n$$ to smaller sample sizes, as shown in the figures below, convergence to the inverse CDF is extremely poor.

Are there bias tricks, sampling or estimation techniques, or modifications to the CDF (i.e. a different weighting scheme than $$\frac{1}{n}, \frac{2}{n}, \dots, \frac{n}{n}$$), that allows us to work confidently with small sample sizes? How can we improve the convergence of a small sample size to the inverse CDF?

Python code:

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm

for n in [1000, 100, 20]:
plt.figure()
plt.title("sample size={}".format(n))
plt.plot(norm.ppf(np.linspace(0, 1, n)), label="invcdf")
plt.plot(np.sort(np.random.normal(size=n)), label="sortsample")
plt.legend()
plt.show()

• Poor only by the unrealistic standards of expecting a small sample to be a miniature of its parent distribution. On a point of detail, writing $1/n, ..., n/n$ implies that a sample can return the maximum possible value of the quantile function, which is impossible for an infinite upper bound. For this reason, sample quantiles are best considered referred to say $(i - a) / (n - 2a + 1)$ for rank $i$ and sample size $n$, with small print disagreements on what $a$ should be. $a = 0.5$ was perhaps the earliest suggestion (Francis Galton). Commented Dec 7, 2020 at 14:09
• If the cdf $F$ is strictly increasing and differentiable at the quantile $F^{-1}(p)$ , with derivative $f$, a CLT applies$$\sqrt{n}(F_n^-(p)-F^{-1}(p))\stackrel{\mathcal L}{\longrightarrow}\mathcal N(0,p(1-p)/f^2(F^{-1}(p)))$$ Commented Dec 7, 2020 at 18:46
• I guess you're trying to say that as sample size gets higher, the distribution converges to Normal. How about the other way around, any comments on how to improve statistical models that can only depend on small sample size? Commented Dec 7, 2020 at 23:11
• stats.stackexchange.com/q/15891/17230 may be of interest. Commented Dec 11, 2020 at 19:34

In general, for samples of small or moderate size, a histogram (even with optimal binning and plotted on a 'density scale') need not give a good approximation of the density function of the population. By contrast, an empirical CDF (ECDF), such as you are using, often gives a more reliable view of the population CDF.

As @NickCox has commented, it is not realistic to expect small samples to give good estimates of CDFs, density functions, or population parameters. In particular, as you have seen, a random sample of size $$n = 20$$ does not often give useful estimates of characteristics of the population.

Suppose you are interested in a population that happens to be $$\mathsf{Norm}(\mu = 100, \sigma = 15).$$ Let's look at a sample of size $$n= 50.$$ Then 95% CIs for $$\mu$$ and $$\sigma$$ are $$(94.18,103.50)$$ and $$(13.56, 20.23),$$ respectively. [Sampling and computations in R.]

set.seed(1207)
x = rnorm(50, 100, 15)
ci.mu = mean(x) + qt(c(.025,.975),49)*sd(x)/sqrt(50); ci.mu
[1]  94.17782 103.40368
ci.sg = sqrt(49*var(x)/qchisq(c(.975,.025),49));  ci.sg
[1] 13.55869 20.22656


A histogram of the data, the density function of the population (black) and a kernel density estimator of that density (dotted red) are shown below.

hist(x, prob=T, col="skyblue2", main="n = 50")
rug(x)
lines(density(x), col="red", lty="dotted", lwd=2)


Here are plots of the population CDF and the sample ECDF (red).

curve(pnorm(x, 100, 15), 50, 150, ylab="Density", main="n = 50")
abline(h=0:1, col="green2")
lines(ecdf(x), col="red", lty="dotted")


With a sample of size $$n = 5000,$$ we get a much better view of the population distribution.

set.seed(1207)
y = rnorm(5000, 100, 15)
ci.mu = mean(y) + qt(c(.025,.975),4999)*sd(y)/sqrt(5000); ci.mu
[1]  99.74097 100.57105
ci.sg = sqrt(4999*var(y)/qchisq(c(.975,.025),4999));  ci.sg
[1] 14.68228 15.26937

hist(y, prob=T, col="skyblue2", main="n = 5000")
curve(dnorm(x, 100, 15), add=T, lwd=2)  # syntax requires 'x'
lines(density(y), col="red", lty="dotted", lwd=2)


The population density (black) and the ECDF (red) are difficult to distinguish.

curve(pnorm(x, 100, 15), 50, 150, lwd=2, ylab="Density", main="n = 5000")
abline(h=0:1, col="green2")
lines(ecdf(y), col="red", lty="dotted")


• so there are no bias tricks, sampling or estimation techniques, or modifications to the CDF that can improve a small-sample statistical model? meaning there is no hope in estimating a model that is doomed to have small sample size to begin with, in other words, futile, and the user has too high standards? Commented Dec 7, 2020 at 23:12
• Not saying $n = 50$ observations tell you nothing. But use as large an $n$ is needed to decide issues important to you. Possible waste of time/energy/money to do a study with too small an $n.$ There are 'sample size' procedures for intelligent speculation about how large $n$ needs to be in order to get useful results for your purpose. // In a real expt (instead of a simulation) it might be good enough to know $\mu$ is btw 94 and 104. (For example, If you need $\mu = 120$ for a profitable product, you can scrap plans for this one and try to get a better idea.) Commented Dec 7, 2020 at 23:14

In complement to BruceET's full answer, here are two slides from my undergraduate mathematical statistics course this trimester:

The first slide shows (left) one realisation of an empirical cdf for a Normal sample of size $$n=100$$ when compared with the truth and (right) the variability of an empirical cdf as a random curve for a Normal sample of size $$n=100$$ when repeated 100 times. I use this experiment to tell my students one can learn a lot from the sample itself without making any assumption on its distribution. In other words and contrary to the OP impression, the performances of the empirical cdf are far from being poor.

The second slide shows (left) one realisation of an empirical cdf for a Normal sample of size $$n=100$$ when compared with the truth and with a Normal cdf using the estimated mean; and (right) the range of variability of an empirical cdf for a Normal sample of size $$n=100$$ over 100 repetitions, compared with the variability of a parametric estimate of the cdf. I use this experiment to tell my students that the additional item of information about the functional form of the distribution helps in making inference about this distribution (if not misspecified).

The CLT $$\sqrt{n}(F_n^-(p)-F^{-1}(p))\stackrel{\mathcal L}{\longrightarrow}\mathcal N(0,p(1-p)/f^2(F^{-1}(p)))$$ gives the scale of the variability of the empirical inverse cdf $$F_n^{-1}$$ (that is, the horizontal variability in the above pictures) as the standard deviation at the $$p$$-quantile of the distribution is of order $$\dfrac{\sqrt{p(1-p)}}{\sqrt nf(F^{-1}(p))}$$ In the case of a Normal sample of size $$n=100$$, when $$p=1/2$$, this standard deviation is $$\dfrac{1}{2\times 10\times 1/\sqrt{2\pi}}\approx 0.125$$ which gives a range of $$\pm1/4$$ on the location of the median.

• So to compensate for small sample size, one can replicate the sample 200 times from a matching parametric distribution and use the average of those replications as a reliable model instead of the original empirical sample itself? Commented Dec 12, 2020 at 15:38
• @develarist: no this is not what this means, in the slide the 200 replications are used to demonstrate the variability under controlled conditions, namely when one already know the distribution of the model. In a statistical framework, there is one and only one sample. Repeating this sample by bootstrap provides a measure of variability, but its quality deteriorates as the sample size $n$ decreases. Commented Dec 12, 2020 at 16:42
• (Mentioning bootstrap in my comment above may actually prove confusing as the primary argument supporting bootstrap is exactly the convergence of the empirical cdf to the true cdf. If the approximation by the ecdf is poor, then the bootstrap evaluation will be poor as well.) Commented Dec 12, 2020 at 16:50