15
$\begingroup$

I would like to find a way to show whether the sample quantile is an unbiased estimator of the true quantiles. Let $F$ be strictly increasing with density function $f$. I will define the $p$-th quantile for $0<p<1$ as $Q(p)=F^{-1}(p)$ and the sample quantile as $$\hat{F}_n^{-1}(p)=\inf\{x:\hat{F}_n(x)\geq p\},$$ where $\hat{F}_n(x)$ is the empirical distribution function, given by $$\hat{F}_n(x)=\frac{1}{n}\sum_{i=1}^n I(X_i \leq x).$$ Based on literature I have read, I expect the sample quantile to be biased, but I am having trouble figuring out how to take the expected value of $\hat{F}_n^{-1}(p)$, particularly since it is defined as the infimum of a set. I do know that the expected value of the empirical distribution function is $F(x)$. Any help or references that could guide me would be greatly appreciated!

$\endgroup$
2
  • $\begingroup$ Cross-posted at math.stackexchange.com/q/3378799/321264. $\endgroup$ Oct 3, 2019 at 4:39
  • $\begingroup$ Asymptotically unbiased for $p$th sample quantile (other than max and min), of a continuous distributions with positive density at $p$th population quantile. Asymptotic dist'n of the sample quantile is normally distributed with pop quantile as mean. (Sort of a CLT for 'central' quantiles.) $\endgroup$
    – BruceET
    Oct 5, 2019 at 3:54

2 Answers 2

13
$\begingroup$

Assuming that $X_1, X_2, X_3 \sim \text{IID } F$ the empirical distribution function has a scaled binomial distribution:

$$\hat{F}_n(x) \sim \frac{1}{n} \cdot \text{Bin}(n, F(x)).$$

For a given probability value $0 < p < 1$ we will denote the sample quantile as:

$$\hat{Q} \equiv \hat{Q}_n(p) \equiv \inf \{ x \in \mathbb{R} | \hat{F}_n(x) \geqslant p \}.$$

Since the empirical distribution function $\hat{F}_n$ is non-decreasing and right-continuous, we have the event equivalence $\inf \{ x \in \mathbb{R} | \hat{F}_n(x) \geqslant p \} \leqslant q$ if and only if $\hat{F}_n(q) \geqslant p$. Thus, the distribution function for the sample quantile is:

$$\begin{equation} \begin{aligned} F_{\hat{Q}}(q) = \mathbb{P}(\hat{Q} \leqslant q) = \mathbb{P} \bigg( \inf \{ x \in \mathbb{R} | \hat{F}_n(x) \geqslant p \} \leqslant q \bigg) = \mathbb{P} \big( \hat{F}_n(q) \geqslant p \big). \\[6pt] \end{aligned} \end{equation}$$


In order to look at the bias of the sample quantile as an estimator of the true quantile, we need to look at the expected value of the former. Using a general expectation rule shown here, the exact expected value of this random variable can be written as the integral:

$$\mathbb{E}(\hat{Q}) = \int \limits_{-\infty}^\infty \Big[ \mathbb{I}(q \geqslant 0) - F_{\hat{Q}}(q) \Big] dq = \int \limits_{-\infty}^\infty \Big[ \mathbb{I}(q \geqslant 0) - \mathbb{P} ( \hat{F}_n(q) \geqslant p ) \Big] dq.$$

This integral is complicated, owing to the scaled binomial distribution for $\hat{F}_n$. However, as $n \rightarrow \infty$ we have $\hat{F}_n(q) \rightarrow F(q)$, and so if $F$ is continuous at $q$ then we also have $Q(\hat{F}_n(q)) \rightarrow q$. This gives the asymptotic convergence:

$$\mathbb{E}(\hat{Q}) \rightarrow \int \limits_{-\infty}^\infty \Big[ \mathbb{I}(q \geqslant 0) - \mathbb{I} ( q \geqslant Q(p) ) \Big] dq = \int \limits_{0}^{Q(p)} dq = Q(p),$$

so long as $F$ is continuous at $p$. Thus, you should expect the sample quantiles to be asymptotically unbiased, except at quantiles corresponding to points of discontinuity of the underlying distribution function. Obviously we may have non-zero bias for finite samples, with the bias depending on the form of the underlying distribution.

$\endgroup$
2
  • 2
    $\begingroup$ (+1) Right about bias for small samples. Simulation in R for $10^6$ samples of size $n=5$ from $\mathsf{Exp}(1):$ h = replicate(10^6, median(rexp(5, 1))); mean(h) returns 0.783058, but population median is 0.6931472. $\endgroup$
    – BruceET
    Oct 5, 2019 at 4:03
  • 1
    $\begingroup$ @BruceET: an exception being the median of a symmetric distribution: for instance, mean(replicate(1e6,median(rnorm(5)))) returns -0.001093016 when mean(replicate(1e6,mean(rnorm(5)))) returns -0.0001424392. $\endgroup$
    – Xi'an
    Apr 29, 2020 at 14:01
3
$\begingroup$

I do not think Ben's derivation is completely correct. The asymptotic unbiasedness feature of sample quantile is not distribution free. There is an important assumption that the r.v. has to satisfy: there is a unique solution $x$ to the condition $F(x-) \leq p \leq F(x)$. A counter example:

Let $X_1, \dots, X_n \sim$ $X$ i.i.d. where $X$ is Bernoulli taking $-1$ with probability $0.5$ and $1$ with probability $0.5$. Now let $p = 0.5$, then the theoretical quantile (median) $Q(p)$ should be $-1$. Now for the sample quantile $$ Q_n(p) = \begin{cases} -1, & \sum_{i=1}^n\mathbf{1}_{X_i = -1} \geq n/2 \\ 1, & \sum_{i=1}^n\mathbf{1}_{X_i = -1} < n/2. \end{cases} $$ Therefore, $$ E(Q_n(p)) = -P\left(\frac{\sum_{i=1}^n\mathbf{1}_{X_i = -1}}{n} \geq 1/2\right) + P\left(\frac{\sum_{i=1}^n\mathbf{1}_{X_i = -1}}{n} < 1/2\right). $$ Now if we take the limit, because of CLT, $$ \lim_{n\to\infty}P\left(\frac{\sum_{i=1}^n\mathbf{1}_{X_i = -1}}{n} \geq 1/2\right) = \lim_{n\to\infty}P\left(\frac{\sum_{i=1}^n\mathbf{1}_{X_i = -1}}{n} < 1/2\right) = 0.5. $$ Therefore, $\lim_{n\to\infty}E(Q_n(p)) = 0 \neq -1$.

In fact, we can also check this with mean(2*(replicate(2000, mean(2*rbernoulli(10^6)-1 ==-1)) >= 0.5)-1) which gave an answer of -0.011. The issue is when the solution to $F(x-) \leq p \leq F(x)$ is not unique, i.e., there are segments where the cdf are flat. The sample quantile will jump around and does not settle down.

$\endgroup$
4
  • $\begingroup$ Well-spotted (+1) --- I have amended my answer to add the required continuity condition. $\endgroup$
    – Ben
    Oct 5, 2020 at 20:21
  • $\begingroup$ Actually I shouldn't say continuous (I corrected it). It should be there is only a unique solution $x$ to the condition $F(x-) \leq p \leq F(x)$. Because you can also have $X\sim unif(0,1)\cup(2,3)$ and pick $p = 0.5$. This is similar to the counterexample I gave. The true median is 1 but the asymptotic expectiation of sample quantile go to $1.5$. But $X$ here still has continuous cdf everywhere. $\endgroup$ Oct 5, 2020 at 21:57
  • $\begingroup$ @NamelessGods In your comment example, I would say the median is any value from $1$ through to $2$, including $1.5$, and in your main example the median is any value from $-1$ through to $+1$, including $0$ $\endgroup$
    – Henry
    Oct 6, 2020 at 8:13
  • $\begingroup$ I don't think it is any value in (-1,1) as proven in my main post.The sample median in my main post is either -1 or 1. But the asymptotic median is certainly 0. $\endgroup$ Oct 6, 2020 at 20:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.