MSE of the Jackknife Estimator for the Uniform distribution

Question

The Jackknife is a resampling method, a predecessor of the Bootstrap, which is useful for estimating the bias and variance of a statistic. This can also be used to apply a "bias correction" to an existing estimator.

Given the estimand $\theta$ and an estimator $\hat\theta \equiv \hat\theta(X_1, X_2, \cdots X_n)$, the Jackknife estimator (with respect to $\hat\theta$) is defined as $$\hat\theta_J = \hat\theta + (n-1)\left(\hat\theta - \frac{1}{n}\sum_{i=1}^n\hat\theta_{-i}\right),$$

where the $\hat\theta_{-i}$ terms denote the estimated value ($\hat\theta$) after "holding out" the $i^{th}$ observation.

---

Let $X_1, X_2, \cdots X_n \stackrel{\text{iid}}{\sim} \text{Unif}(0, \theta)$ and consider the estimator $\hat\theta = X_{(n)}$ (i.e. the maximum value, also the MLE). Note that

$$\hat\theta_{-i} = \begin{cases} X_{(n-1)}, & X_i = X_{(n)} \\[1.2ex] X_{(n)}, & X_i \neq X_{(n)} \end{cases}.$$

Thus the Jackknife estimator here can be written as a linear combination of the two largest values

\begin{align*} \hat\theta_J &= X_{(n)} + \frac{n-1}{n}\left(X_{(n)} - X_{(n-1)}\right) \\[1.3ex] &= \frac{2n-1}{n}X_{(n)} - \frac{n-1}{n} X_{(n-1)}. \end{align*}

What is the bias, variance and mean square error ?

Answer 1

It is well known that the order statistics, sampled from a uniform distribution, are Beta-distributed random variables (when properly scaled). $$\frac{X_{(j)}}{\theta} \sim \text{Beta}(j, n+1-j)$$ Using standard properties of the Beta distribution we can obtain the mean and variance of $X_{(n)}$ and $X_{(n-1)}$.

Bias

\begin{align*} E\left(\hat\theta_J\right) &= \frac{2n-1}{n}E(X_{(n)}) - \frac{n-1}{n}E(X_{(n-1)}) \\[1.3ex] &= \frac{2n-1}{n}\frac{n}{n+1}\theta - \frac{n-1}{n}\frac{n-1}{n+1}\theta \\[1.3ex] &= \frac{n(n+1) - 1}{n(n+1)} \theta \end{align*}

Therefore the bias of $\hat\theta_J$ is given by $$\text{Bias}_\theta(\hat\theta_J) = \frac{-\theta}{n(n+1)}$$

Variance

Note: Covariance is derived here. \begin{align*} \text{Var}\left(\hat\theta_J\right) &= \frac{(2n-1)^2}{n^2}\text{Var}(X_{(n)}) + \frac{(n-1)^2}{n^2}\text{Var}(X_{(n-1)}) - 2 \frac{2n-1}{n}\frac{n-1}{n}\text{Cov}(X_{(n)}, X_{(n-1)}) \\[1.3ex] &= \frac{(2n-1)^2}{n^2}\frac{n\theta^2}{(n+1)^2(n+2)} + \frac{(n-1)^2}{n^2}\frac{2(n-1)\theta^2}{(n+1)^2(n+2)} - \frac{2(2n-1)(n-1)}{n^2}\frac{(n-1)\theta^2}{(n+1)^2(n+2)} \\[1.5ex] \end{align*} $$= \frac{(2n^2-1)\theta^2}{n(n+1)^2(n+2)}$$

MSE

Using the decomposition $\text{MSE}_\theta(\hat\theta) = \text{Bias}^2_\theta(\hat\theta) + \text{Var}(\hat\theta)$, we have

\begin{align*} \text{MSE}_\theta(\hat\theta_J) &= \left(\frac{-\theta}{n(n+1)}\right)^2 + \frac{(2n^2-1)\theta^2}{n(n+1)^2(n+2)} \\[1.3ex] &= \frac{2(n-1+1/n)\theta^2}{n(n+1)(n+2)}\\[1.3ex] &= \mathcal O(n^{-2}) \end{align*}