How to prove $s^2$ is a consistent estimator of $\sigma^2$?

Question

I am trying to prove that $s^2=\frac{1}{n-1}\sum^{n}_{i=1}(X_i-\bar{X})^2$ is a consistent estimator of $\sigma^2$ (variance), meaning that as the sample size $n$ approaches $\infty$ , $\text{var}(s^2)$ approaches 0 and it is unbiased.

I understand how to prove that it is unbiased, but I cannot think of a way to prove that $\text{var}(s^2)$ has a denominator of n. Does anyone have any ways to prove this?

---

The question asks:

A random sample of size n is taken from a normal population with variance $\sigma^2$. Show that the statistic $s^2$ is a consistent estimator of $\sigma^2$

So far I have gotten:
$\text{var}(s^2) = \text{var}(\frac{1}{n-1}\Sigma X^2-n\bar X^2)$
$= \frac{1}{(n-1)^2}(\text{var}(\Sigma X^2) + \text{var}(n\bar X^2))$
$= \frac{n^2}{(n-1)^2}(\text{var}(X^2) + \text{var}(\bar X^2))$

But as I do not know how to find $Var(X^2) $and$ Var(\bar X^2)$, I am stuck here (I have already proved that $S^2$ is an unbiased estimator of $Var(\sigma^2)$)

Source : Edexcel AS and A Level Modular Mathematics S4 (from 2008 syllabus) Examination Style Paper Question 1. This is for my own studies and not school work.

Answer 1

It's a very well known result that :

If $X_1, X_2, \cdots, X_n \stackrel{\text{iid}}{\sim} N(\mu,\sigma^2)$ , then $$Z_n = \dfrac{\displaystyle\sum(X_i - \bar{X})^2}{\sigma^2} \sim \chi^2_{n-1}$$ Thus, $ \mathbb{E}(Z_n) = n-1 $ and $ \text{var}(Z_n) = 2(n-1)$ .

If you wish to see a proof of the above result, please refer to this link.

Now, since you already know that $s^2$ is an unbiased estimator of $\sigma^2$ , so for any $\varepsilon>0$ , we have :

\begin{align*} &\mathbb{P}(\mid s^2 - \sigma^2 \mid > \varepsilon )\\ &= \mathbb{P}(\mid s^2 - \mathbb{E}(s^2) \mid > \varepsilon )\\ &\leqslant \dfrac{\text{var}(s^2)}{\varepsilon^2}\\ &=\dfrac{1}{(n-1)^2}\cdot \text{var}\left[\sum (X_i - \overline{X})^2)\right]\\ &=\dfrac{\sigma^4}{(n-1)^2}\cdot \text{var}\left[\frac{\sum (X_i - \overline{X})^2}{\sigma^2}\right]\\ &=\dfrac{\sigma^4}{(n-1)^2}\cdot\text{var}(Z_n)\\ &=\dfrac{\sigma^4}{(n-1)^2}\cdot 2(n-1) = \dfrac{2\sigma^4}{n-1} \stackrel{n\to\infty}{\longrightarrow} 0 \end{align*}

Thus, $ \displaystyle\lim_{n\to\infty} \mathbb{P}(\mid s^2 - \sigma^2 \mid > \varepsilon ) = 0$ , i.e. $ s^2 \stackrel{\mathbb{P}}{\longrightarrow} \sigma^2 $ as $n\to\infty$ , which tells us that $s^2$ is a consistent estimator of $\sigma^2$ .

---

Note : I have used Chebyshev's inequality in the first inequality step used above. Hope my answer serves your purpose. Thank you.

Answer 2

I have found a much simpler proof using the weak law of large numbers (This requires finite second moment):

$\begin{aligned} \frac{1}{n-1}\sum\left(x_i-\bar{x}_n\right)^2 & =\frac{1}{n-1}\left(\sum x_i^2-2 x_i \bar{x}_n+\bar{x}_n^2\right) \\ & =\frac{1}{n-1}\left(\sum x_i^2-2 \sum x_i \bar{x}_n+\sum \bar{x}_n^2\right) \\ & =\frac{1}{n-1}\left(\sum x_i^2-2 \sum x_i \bar{x}_n+\sum \bar{x}_n^2\right) \\ & =\frac{1}{n-1}\left(\sum x_i^2-2 n \frac{1}{n} \sum x_i \bar{x}_n+\sum \bar{x}_n^2\right) \\ & =\frac{1}{n-1}\left(\sum x_i^2-2 n \bar{x}_n^2+\sum \bar{x}_n^2\right) \\ & =\frac{1}{n-1} \left(\sum x_i^2-2 \sum \bar{x}_n^2+\sum \bar{x}_n^2\right) \\ & =\frac{1}{n-1}\left(\sum x_i^2-\sum \bar{x}_n^2\right) \\ & =\frac{n}{n-1}\left(\frac{1}{n} \sum x_i^2-\bar{x}_n^2\right) \stackrel{P}{\longrightarrow} 1 \cdot\left(E\left[x_1^2\right]-E\left[x_1\right]^2\right)=\sigma^2.\end{aligned}$

And thus by the definition of consistency $s^2$ is a consistent estimator of the variance.

Answer 3

Vanishing variance (and resulting convergence in mean square) occurs if the underlying distribution has finite kurtosis

The other answer here considers the case of a sample variance of IID normally distributed random variables, but this is only one special case which rarely holds for most cases of interest. I will proceed for a more general case and look at convergence in mean-square. Your question is a bit ambiguous about what type of convergence you are interested in, but convergence in mean-square implies convergence in probability so this will hopefully be a useful result for you.

We can proceed for the more general case by considering IID random variables from any underlying distribution with variance $\sigma^2$ and kurtosis $\kappa$. Given $n$ sample values, the variance of the sample variance (see e.g., O'Neill 2014, p. 284) is:

$$\mathbb{V}(S_n^2) = \Big( \kappa - \frac{n-3}{n-1} \Big) \frac{\sigma^4}{n}.$$

(Please have a look at the linked paper if you would like to see the derivation of this result.) Now, if the kurtosis of the underlying distribuion is infinite ($\kappa = \infty$) then the variance of the sample variance is also infinite for all $n$ and you won't get convergence in mean-square for the sample variance. However, if the kurtosis is finite ($\kappa < \infty$) then we get:

$$\begin{align} \lim_{n \rightarrow \infty} \mathbb{V}(S_n^2) &= \lim_{n \rightarrow \infty} \Big( \kappa - \frac{n-3}{n-1} \Big) \frac{\sigma^4}{n} \\[6pt] &= (\kappa - 1) \sigma^4 \lim_{n \rightarrow \infty} \frac{1}{n} \\[12pt] &= 0. \\[6pt] \end{align}$$

Since the sample variance is an unbiased estimator of $\sigma^2$, this is sufficient to show that the sample variance converges in mean-square (and therefore also converges in probability) to $\sigma^2$. This means that the sample variance converges to the true variance for IID data, so long as the underlying distribution has finite kurtosis.