What's an example of an estimator that is $o_p(n^{-1/2})$?

Question

For $X_i$ i.i.d. normal with mean $\mu < \infty$ and variance 1, by the law of large numbers, we have $\bar{X} \overset{\mathcal{P}}{\rightarrow}\mu$, i.e. $\bar{X} - \mu = o_p(1)$. Similarly, by CLT, we have that $\bar{X} - \mu = O_p(n^{-1/2})$ because $\sqrt{n}(\bar{X} - \mu) \overset{\mathcal{D}}{\rightarrow} N(0, 1).$

Can anyone give me an example of an estimator that is $o_p(n^{-1/2})$? From what I understand, it's an estimator $\hat{\theta}$ such that $\sqrt{n} \hat{\theta} \overset{\mathcal{P}}{\rightarrow} 0,$ correct?

Answer 1

Estimators of percentiles can act like this.

The idea is that when a great deal of probability is concentrated in small neighborhoods around a percentile, then the sample percentile will tend to come extremely close to the true percentile. (Indeed, an extreme case occurs when a distribution has an atom at a percentile, because for sufficiently large samples it is more and more likely that the sample percentile will exactly equal the distribution percentile.)

---

Rather than get too involved with the analysis, let me present one of the simpler examples of estimating medians, using my post at Central Limit Theorem for Sample Medians as a point of departure. There I consider a distribution $F$ for a sample of size $n$ (assumed to be an even number for convenience) and introduce the Beta$(n/2+1,n/2+1)$ distribution to describe the sample median.

To translate that setting to this one, I propose estimating the median $\tilde \mu$ of $F$ by means of the sample median. If we let $1/2-q_\alpha$ be the lower $100\alpha/2$ percentile of $G$ and $1/2+q_\alpha$ its upper $100\alpha/2$ percentile, then with probability at least $1-\alpha,$ the sample median will lie between $F^{-1}(1/2-q_\alpha)$ and $F^{-1}(1/2+q_\alpha).$

There exist distributions where, given sufficiently small $\epsilon\gt 0,$ for all $q$ with $1/2-\epsilon\lt q \lt 1/2+\epsilon,$

$$F^{-1}(q) = C \operatorname{sgn}(q) |q|^p.$$

Here, $C$ is a positive constant and $p \gt 0.$ (Indeed, this describes the behavior of "most" distributions, but typically $p=1.$) Thus, on the interval $(F^{-1}(1/2-\epsilon), F^{-1}(1/2+\epsilon)),$ $F$ has a density $f.$ When $p\gt 1,$ $f$ diverges at $F^{-1}(1/2)$ because the graph of $F$ becomes vertical there. This signals the special behavior sought in the question--but it remains to analyze what happens.

Here are some examples showing graphs of such $F$, labeled by the values of $p.$ The case $p=\infty$ corresponds to an $F$ for which $F^{-1}(1/2+q)$ approaches the median faster than any positive power of $q$ as $q\to 0.$ This particular $F$ is given by

$$F^{-1}(1/2+q) = \operatorname{sgn}(q) \exp\left(1 - \frac{1}{|2q|}\right).$$

These kinds of distributions serve as natural models of the "peakedness" of any distribution at a specified percentile, and as such would have applications in finance, natural systems, and elsewhere: they are not mere mathematical curiosities or "pathologies."

Because the mean of $G$ is $1/2,$ its variance is asymptotically $1/n,$ and it asymptotically is Normal, we conclude that when $n \gg (Z_{\alpha/2} / \epsilon) ^{2},$

$$q_\alpha \lt n^{-1/2}$$

where $Z_{\alpha/2} = \Phi^{-1}(1-\alpha/2)$ is a percentile of the standard Normal distribution. As a result,

$$F^{-1}(1/2+q_\alpha) = C |q_\alpha|^p \lt C n^{-p/2}.$$

This shows that for such distributions,

The sample median approaches the true median in probability at a rate no slower than $n^{-p/2}.$ Selecting $p\gt 1$ gives the desired example (because $O(n^{-p/2})=o(n^{-1/2})$.

---

To illustrate, consider the distribution functions defined on $x\in [-1,1]$ by

$$F_p(x) = \frac{1}{2}\left(1 + \operatorname{sgn}(x) |x|^{1/p}\right).$$

I simulated samples of size $n=10$ through $n=10000,$ with $500$ iterations of each simulation, to estimate the standard deviation of the sample median. The foregoing assertion is tantamount to claiming that, on log-log axes, the plot of the SD against the sample size has a slope of $-p/2$ when $p$ is finite. (When $p$ is infinite, the curve never becomes linear, but just keeps dropping faster and faster.) This simulation bears out that claim:

Finally, with more detailed analysis it is possible to control the shape of the peak in great detail. This furnishes examples where the estimator does not behave asymptotically according to any law at all: as $n$ grows ever larger, the estimator can bounce around various regimes of asymptotic behavior and never settle down to any definite rate as a function of $n,$ even though (of course) it will be converging towards the true percentile.

For those interested in the details, here is the R code for the simulation and the last figure.

#
# Generate random variates for a distribution with median 0 and "peakedness"
# of order `p` there.
#
rf <- function(n, p=1) {
  u <- runif(n, -1, 1)
  if (is.infinite(p)) {
    sign(u) * exp(1 - abs(1/u))
  } else {
    sign(u) * abs(u)^p
  }
}
#
# Simulate from some of these distributions to estimate the standard deviation
# of the sample median.  This will take a few seconds.
#
n.sim <- 500
n <- ceiling(10^(seq(1, 4, length.out=9)))
l.X <- lapply (c(1,2,3,Inf), function(p) {
  s <- sapply(n, function(n) {
    x <- apply(matrix(rf(n.sim*n, p), nrow=n), 2, median)
    sd(x)
  })
  data.frame(n=n, sd=s, p=p, n.sim=n.sim)
})
X <- do.call(rbind, l.X)
#
# Plot the results.
#
X$p <- factor(X$p)
library(ggplot2)
ggplot(X, aes(n, sd)) + 
  scale_x_log10() + scale_y_log10() + 
  geom_smooth(aes(col=p), se=FALSE, span=.9) + 
  geom_point(aes(fill=p), pch=21)

Answer 2

The range for $n$ i.i.d. uniform random variables on $[0, 1]$ has a $\mbox{Beta}(n-1, 2)$ distribution, thus its variance is $\frac{2(n-1)}{(n+1)^2 (n+2)}$. Thus this estimator is $O(n^{-1})$ and thus $o(n^{-1/2})$.

At the other extreme, the range for $n$ i.i.d. exponential random variables has a variance equal to $\frac{1}{\lambda^2}\sum_{k=1}^{n-1}\frac{1}{k^2}$. Thus the variance converges to $\frac{\pi^2}{6\lambda^2}$. Thus it is $O(1)$.

These results are discussed (with proof) in my recent article A Beautiful Result in Probability Theory.

Answer 3

Here's an example that doesn't involve quantiles/extremes:

Suppose $(X,Y)$ are bivariate Normal with correlation $\rho$ and consider the usual estimator $$\hat\rho=\frac{\sum_i (X_i-\bar X) (Y_i-\bar Y)}{\sqrt{\sum_i (X_i-\bar X)^2\sum_j(Y_j-\bar Y)^2}}$$

Now suppose we want $\rho^2$ and we estimate it with $\hat\rho^2$. If $\rho\neq 0$, $$\sqrt{n}(\hat\rho^2-\rho^2)\stackrel{d}{\to} N(0,\tau^2_{\rho})$$ for some complicated $\tau^2_{\rho}$, by the delta method.

But if $\rho=0$, $\partial \rho^2/\partial \rho=0$ and the delta- method tells us $$\sqrt{n}(\hat\rho^2-\rho^2)\stackrel{p}{\to} 0$$ and $\hat\rho=o_p(n^{-1/2})$