Is the sample quantile unbiased for the true quantile?

Question

I would like to find a way to show whether the sample quantile is an unbiased estimator of the true quantiles. Let $F$ be strictly increasing with density function $f$. I will define the $p$-th quantile for $0<p<1$ as $Q(p)=F^{-1}(p)$ and the sample quantile as $$\hat{F}_n^{-1}(p)=\inf\{x:\hat{F}_n(x)\geq p\},$$ where $\hat{F}_n(x)$ is the empirical distribution function, given by $$\hat{F}_n(x)=\frac{1}{n}\sum_{i=1}^n I(X_i \leq x).$$ Based on literature I have read, I expect the sample quantile to be biased, but I am having trouble figuring out how to take the expected value of $\hat{F}_n^{-1}(p)$, particularly since it is defined as the infimum of a set. I do know that the expected value of the empirical distribution function is $F(x)$. Any help or references that could guide me would be greatly appreciated!

Answer 1

Assuming that $X_1, X_2, X_3 \sim \text{IID } F$ the empirical distribution function has a scaled binomial distribution:

$$\hat{F}_n(x) \sim \frac{1}{n} \cdot \text{Bin}(n, F(x)).$$

For a given probability value $0 < p < 1$ we will denote the sample quantile as:

$$\hat{Q} \equiv \hat{Q}_n(p) \equiv \inf \{ x \in \mathbb{R} | \hat{F}_n(x) \geqslant p \}.$$

Since the empirical distribution function $\hat{F}_n$ is non-decreasing and right-continuous, we have the event equivalence $\inf \{ x \in \mathbb{R} | \hat{F}_n(x) \geqslant p \} \leqslant q$ if and only if $\hat{F}_n(q) \geqslant p$. Thus, the distribution function for the sample quantile is:

$$\begin{equation} \begin{aligned} F_{\hat{Q}}(q) = \mathbb{P}(\hat{Q} \leqslant q) = \mathbb{P} \bigg( \inf \{ x \in \mathbb{R} | \hat{F}_n(x) \geqslant p \} \leqslant q \bigg) = \mathbb{P} \big( \hat{F}_n(q) \geqslant p \big). \\[6pt] \end{aligned} \end{equation}$$

---

In order to look at the bias of the sample quantile as an estimator of the true quantile, we need to look at the expected value of the former. Using a general expectation rule shown here, the exact expected value of this random variable can be written as the integral:

$$\mathbb{E}(\hat{Q}) = \int \limits_{-\infty}^\infty \Big[ \mathbb{I}(q \geqslant 0) - F_{\hat{Q}}(q) \Big] dq = \int \limits_{-\infty}^\infty \Big[ \mathbb{I}(q \geqslant 0) - \mathbb{P} ( \hat{F}_n(q) \geqslant p ) \Big] dq.$$

This integral is complicated, owing to the scaled binomial distribution for $\hat{F}_n$. However, as $n \rightarrow \infty$ we have $\hat{F}_n(q) \rightarrow F(q)$, and so if $F$ is continuous at $q$ then we also have $Q(\hat{F}_n(q)) \rightarrow q$. This gives the asymptotic convergence:

$$\mathbb{E}(\hat{Q}) \rightarrow \int \limits_{-\infty}^\infty \Big[ \mathbb{I}(q \geqslant 0) - \mathbb{I} ( q \geqslant Q(p) ) \Big] dq = \int \limits_{0}^{Q(p)} dq = Q(p),$$

so long as $F$ is continuous at $p$. Thus, you should expect the sample quantiles to be asymptotically unbiased, except at quantiles corresponding to points of discontinuity of the underlying distribution function. Obviously we may have non-zero bias for finite samples, with the bias depending on the form of the underlying distribution.

Answer 2

I do not think Ben's derivation is completely correct. The asymptotic unbiasedness feature of sample quantile is not distribution free. There is an important assumption that the r.v. has to satisfy: there is a unique solution $x$ to the condition $F(x-) \leq p \leq F(x)$. A counter example:

Let $X_1, \dots, X_n \sim$ $X$ i.i.d. where $X$ is Bernoulli taking $-1$ with probability $0.5$ and $1$ with probability $0.5$. Now let $p = 0.5$, then the theoretical quantile (median) $Q(p)$ should be $-1$. Now for the sample quantile $$ Q_n(p) = \begin{cases} -1, & \sum_{i=1}^n\mathbf{1}_{X_i = -1} \geq n/2 \\ 1, & \sum_{i=1}^n\mathbf{1}_{X_i = -1} < n/2. \end{cases} $$ Therefore, $$ E(Q_n(p)) = -P\left(\frac{\sum_{i=1}^n\mathbf{1}_{X_i = -1}}{n} \geq 1/2\right) + P\left(\frac{\sum_{i=1}^n\mathbf{1}_{X_i = -1}}{n} < 1/2\right). $$ Now if we take the limit, because of CLT, $$ \lim_{n\to\infty}P\left(\frac{\sum_{i=1}^n\mathbf{1}_{X_i = -1}}{n} \geq 1/2\right) = \lim_{n\to\infty}P\left(\frac{\sum_{i=1}^n\mathbf{1}_{X_i = -1}}{n} < 1/2\right) = 0.5. $$ Therefore, $\lim_{n\to\infty}E(Q_n(p)) = 0 \neq -1$.

In fact, we can also check this with mean(2*(replicate(2000, mean(2*rbernoulli(10^6)-1 ==-1)) >= 0.5)-1) which gave an answer of -0.011. The issue is when the solution to $F(x-) \leq p \leq F(x)$ is not unique, i.e., there are segments where the cdf are flat. The sample quantile will jump around and does not settle down.