# Asymptotic distribution of "plug-in" variance estimator for Bernoulli data

I am stuck at the following exercise:

Let $$(X_k)_k$$ be a sequence of i.i.d. Bernoulli$$(p)$$ distributed RVs with $$p=1/2$$. We want to estimate $$Var(X_1) = p(1-p) = p^2 =: \tau_p^2$$ by directly plugging in the natural estimator $$\widetilde{p} := \frac{1}{n}\sum_{i=1}^n X_i$$ of $$p$$. We call this estimator for the variance $$\widetilde{\tau}^2_{p,n}$$. Examine the limit distribution of $$\widetilde{\tau}^2_{p,n}$$.

I do not see how I could do this. I recognise that we are basically looking at a uniform distribution of two events and I see that $$\widetilde{\tau}^2_{p,n}$$ has a rather nice formula

$$\widetilde{\tau}^2_{p,n} = \frac{1}{n^2} \biggl(\sum_{i=1}^n X_i \biggr) ^2 = \overline{X}^2,$$

but I do not see how to proceed now. Intuitively I would say that $$\widetilde{\tau}^2_{p,n}$$ should also be uniformly distributed, but how can I prove this?

• To see that your intuition is incorrect, consider that the distribution of $\tilde p$ must be $1/n$ times a Binomial$(n,p)$ distribution, which for $n\gt 1$ is far from uniform, suggesting (correctly) the distribution of $\tilde{\tau}^2_{p,n}$ is nonuniform, too. BTW, the limit distribution of $\tilde{\tau}^2_{p,n}$ is the constant $1/4$ -- but that's a trivial and almost worthless observation. The question likely wants you to standardize the distribution.
– whuber
Nov 10, 2020 at 22:57
• Could you use a Monte Carlo simulation? Nov 10, 2020 at 23:03

Firstly, let's start by fixing your estimator. Although $$p(1-p)=p^2$$ in the special case where $$p=\tfrac{1}{2}$$, in the context of parameter estimation, we don't know the parameter $$p$$, and so we don't know that $$p(1-p)=p^2$$. Consequently, the plug-in variance estimator is $$\tilde{\tau}^2_{n} = \tilde{p}_n (1-\tilde{p}_n)$$, not $$\tilde{p}_n^2$$. I will show an asymptotic approximation derived using the central limit theorem to approximate the true distribution function for the estimator. This is quite a tricky problem, and it has a few parts, but it leads to quite a useful asymptotic form.

Let's start by writing the variance estimator out in terms of the number of "successes" in the underlying Bernoulli random variables. Let $$K_n \equiv \sum_{i=1}^n X_i$$ denote the number of successes. Then we can write the plug-in estimator as:

$$\tilde{\tau}^2_{n} = \tilde{p}_n (1-\tilde{p}_n) = \frac{1}{n^2} (n K_n - K_n^2).$$

We will also note that, in the present case where $$p=\tfrac{1}{2}$$, we can use the central limit theorem to obtain the asymptotic distribution:

$$Z_n \equiv 2 \sqrt{n} \Bigg( \frac{K_n}{n} - \frac{1}{2} \Bigg) \sim \text{N}(0,1).$$

The range of possible values for the variance estimator is $$0 \leqslant \tilde{\tau}^2_{n} \leqslant \tfrac{1}{4}$$. For all $$0 \leqslant t \leqslant \tfrac{1}{4}$$, the cumulative distribution function for this estimator is given by:

\begin{align} \mathbb{P}(\tilde{\tau}^2_{n} \leqslant t) &= \mathbb{P}(n K_n - K_n^2 \leqslant t n^2) \\[14pt] &= \mathbb{P}(K_n^2 - n K_n + t n^2 \geqslant 0) \\[8pt] &= 1 - \mathbb{P} \Bigg( \frac{1 - \sqrt{1-4t}}{2} < \frac{K_n}{n} < \frac{1 + \sqrt{1-4t}}{2} \Bigg) \\[6pt] &= 1 - \mathbb{P} \Bigg( - \frac{\sqrt{1-4t}}{2} < \frac{K_n}{n} - \frac{1}{2} < \frac{\sqrt{1-4t}}{2} \Bigg) \\[6pt] &= 1 - \mathbb{P} \Bigg( - \sqrt{(1-4t)n} < 2 \sqrt{n} \Bigg( \frac{K_n}{n} - \frac{1}{2} \Bigg) < \sqrt{(1-4t)n} \Bigg) \\[8pt] &= 1 - \mathbb{P} \Big( - \sqrt{(1-4t)n} < Z_n < \sqrt{(1-4t)n} \Big). \\[6pt] \end{align}

(The third step in this equation follows from re-arrangement of the quadratic inequality; full working shown below.) When $$n$$ is large we have the asymptotic approximation $$Z_n \sim \text{N}(0,1)$$ so we get the asymptotic approximation $$\mathbb{P}(\tilde{\tau}^2_{n} \leqslant t) \approx F_\text{asymp}(t)$$ given by:

$$F_\text{asymp}(t) = 1 - \frac{\Phi ( \sqrt{(1-4t)n} ) - \Phi ( -\sqrt{(1-4t)n} )}{\Phi (\sqrt{n}) - \Phi (-\sqrt{n})}.$$

(Note that I have added a denominator here to obtain a distribution function that is valid for all $$n$$ over the range $$0 \leqslant t \leqslant \tfrac{1}{4}$$. This form ensures that we get zero probability when $$t=0$$ and unit probability when $$t = \tfrac{1}{4}$$.) Differentiating then gives the asymptotic density:

\begin{align} f_\text{asymp}(t) &= \sqrt{\frac{n}{2 \pi (1-4t)}} \cdot \frac{\exp ( - (1-4t)n/2 )}{\Phi (\sqrt{n}) - \Phi (-\sqrt{n})} \cdot \mathbb{I}(0 \leqslant t \leqslant \tfrac{1}{4}) \\[6pt] &= \frac{\mathbb{I}(0 \leqslant 1-4t \leqslant 1)}{\Phi (\sqrt{n}) - \Phi (-\sqrt{n})} \cdot \text{Ga} \Big( 1-4t \ \Big| \ \text{Shape} = \frac{1}{2}, \ \text{Rate} = \frac{n}{2} \Big). \\[6pt] \end{align}

This means that our estimator has an asymptotic distribution that is a truncated gamma distribution:

$$1 - 4 \tilde{\tau}^2_{n} \overset{\text{Asymp}}{\sim} \text{TruncGamma} \Big( \text{Shape} = \frac{1}{2}, \ \text{Rate} = \frac{n}{2}, \ \text{Upper Bound} = 1 \Big).$$

This gives you an asymptotic density function based on approximating the binomial random variable $$K_n$$ using the central limit theorem. The asymptotic form is a continuous approximation to a discrete distribution, but it should give quite a good approximation to the true distribution when $$n$$ is large.

Working for the quadratic inequality: Using quadratic formula we can find the roots of the quadratic form shown in the inequality, which are:

$$r = n \cdot \frac{1 \pm \sqrt{1-4t}}{2}.$$

Consequently, we can write the quadratic form as:

$$K_n^2 - n K_n + t n^2 = \Big(K_n - n \cdot \frac{1 - \sqrt{1-4t}}{2} \Big) \Big( K_n - n \cdot \frac{1 + \sqrt{1-4t}}{2} \Big).$$

The quadratic inequality $$K_n^2 - n K_n + t n^2 \geqslant 0$$ occurs when both terms are non-negative, or both terms are non-positive. This excludes the range:

$$\frac{1 - \sqrt{1-4t}}{2} < \frac{K_n}{n} < \frac{1 + \sqrt{1-4t}}{2}.$$