12
$\begingroup$

I am trying to prove that $s^2=\frac{1}{n-1}\sum^{n}_{i=1}(X_i-\bar{X})^2$ is a consistent estimator of $\sigma^2$ (variance), meaning that as the sample size $n$ approaches $\infty$ , $\text{var}(s^2)$ approaches 0 and it is unbiased.

I understand how to prove that it is unbiased, but I cannot think of a way to prove that $\text{var}(s^2)$ has a denominator of n. Does anyone have any ways to prove this?


The question asks:

A random sample of size n is taken from a normal population with variance $\sigma^2$. Show that the statistic $s^2$ is a consistent estimator of $\sigma^2$

So far I have gotten:
$\text{var}(s^2) = \text{var}(\frac{1}{n-1}\Sigma X^2-n\bar X^2)$
$= \frac{1}{(n-1)^2}(\text{var}(\Sigma X^2) + \text{var}(n\bar X^2))$
$= \frac{n^2}{(n-1)^2}(\text{var}(X^2) + \text{var}(\bar X^2))$

But as I do not know how to find $Var(X^2) $and$ Var(\bar X^2)$, I am stuck here (I have already proved that $S^2$ is an unbiased estimator of $Var(\sigma^2)$)

Source : Edexcel AS and A Level Modular Mathematics S4 (from 2008 syllabus) Examination Style Paper Question 1. This is for my own studies and not school work.

$\endgroup$
7
  • 1
    $\begingroup$ Please add the [self-study] tag & read its wiki. Also, you can use $\LaTeX$ markup on this site, see math.meta.stackexchange.com/questions/5020/…. $\endgroup$ Nov 14, 2020 at 9:51
  • 1
    $\begingroup$ The decomposition of the variance is incorrect in several aspects. $\endgroup$
    – Xi'an
    Nov 14, 2020 at 10:55
  • $\begingroup$ @Xi'an On the third line of working, I realised I did not put a ^2 on the n on the numerator of the fraction $\endgroup$ Nov 14, 2020 at 11:02
  • $\begingroup$ @MrDerpinati, please have a look at my answer, and let me know if it's understandable to you or not. $\endgroup$
    – JRC
    Nov 14, 2020 at 11:47
  • 1
    $\begingroup$ @Xi'an My textbook did not cover the variation of random variables that are not independent, so I am guessing that if $X_i$ and $\bar X_n$ are dependent, $Var(X_i +\bar X_n) = Var(X_i) + Var(\bar X_n)$ ? $\endgroup$ Nov 14, 2020 at 14:18

3 Answers 3

11
$\begingroup$

It's a very well known result that :

If $X_1, X_2, \cdots, X_n \stackrel{\text{iid}}{\sim} N(\mu,\sigma^2)$ , then $$Z_n = \dfrac{\displaystyle\sum(X_i - \bar{X})^2}{\sigma^2} \sim \chi^2_{n-1}$$ Thus, $ \mathbb{E}(Z_n) = n-1 $ and $ \text{var}(Z_n) = 2(n-1)$ .

If you wish to see a proof of the above result, please refer to this link.

Now, since you already know that $s^2$ is an unbiased estimator of $\sigma^2$ , so for any $\varepsilon>0$ , we have :

\begin{align*} &\mathbb{P}(\mid s^2 - \sigma^2 \mid > \varepsilon )\\ &= \mathbb{P}(\mid s^2 - \mathbb{E}(s^2) \mid > \varepsilon )\\ &\leqslant \dfrac{\text{var}(s^2)}{\varepsilon^2}\\ &=\dfrac{1}{(n-1)^2}\cdot \text{var}\left[\sum (X_i - \overline{X})^2)\right]\\ &=\dfrac{\sigma^4}{(n-1)^2}\cdot \text{var}\left[\frac{\sum (X_i - \overline{X})^2}{\sigma^2}\right]\\ &=\dfrac{\sigma^4}{(n-1)^2}\cdot\text{var}(Z_n)\\ &=\dfrac{\sigma^4}{(n-1)^2}\cdot 2(n-1) = \dfrac{2\sigma^4}{n-1} \stackrel{n\to\infty}{\longrightarrow} 0 \end{align*}

Thus, $ \displaystyle\lim_{n\to\infty} \mathbb{P}(\mid s^2 - \sigma^2 \mid > \varepsilon ) = 0$ , i.e. $ s^2 \stackrel{\mathbb{P}}{\longrightarrow} \sigma^2 $ as $n\to\infty$ , which tells us that $s^2$ is a consistent estimator of $\sigma^2$ .


Note : I have used Chebyshev's inequality in the first inequality step used above. Hope my answer serves your purpose. Thank you.

$\endgroup$
5
  • $\begingroup$ Since the OP is unable to compute the variance of $Z_n$, it is neither well-know nor straightforward for them. I thus suggest you also provide the derivation of this variance. $\endgroup$
    – Xi'an
    Nov 14, 2020 at 12:15
  • $\begingroup$ I feel like I have seen a similar answer somewhere before in my textbook (I couldn't find where!) but the method is very different. As I am doing 11th/12th grade (A Level in the UK) maths, to me, this seems like a university level answer, and thus I do not really understand this. Thank you for your input, but I am sorry to say I do not understand. $\endgroup$ Nov 14, 2020 at 14:07
  • 1
    $\begingroup$ I guess there isn't any easier explanation to your query other than what I wrote. Also, what @Xi'an is talking about surely needs a proof which isn't very elementary (I've mentioned a link). In fact, the definition of Consistent estimators is based on Convergence in Probability. Do you know what that means ? $\endgroup$
    – JRC
    Nov 14, 2020 at 19:59
  • 3
    $\begingroup$ I don't understand why this answer got so many upvotes and accepted -- the normality assumption is not given at all in the original problem. $\endgroup$
    – Zhanxiong
    Nov 23 at 1:42
  • $\begingroup$ This is what happens when few folks already voted and others follow in with superficial reading @Zhanxiong. $\endgroup$ Nov 23 at 2:52
4
$\begingroup$

I have found a much simpler proof using the weak law of large numbers (This requires finite second moment):

$\begin{aligned} \frac{1}{n-1}\sum\left(x_i-\bar{x}_n\right)^2 & =\frac{1}{n-1}\left(\sum x_i^2-2 x_i \bar{x}_n+\bar{x}_n^2\right) \\ & =\frac{1}{n-1}\left(\sum x_i^2-2 \sum x_i \bar{x}_n+\sum \bar{x}_n^2\right) \\ & =\frac{1}{n-1}\left(\sum x_i^2-2 \sum x_i \bar{x}_n+\sum \bar{x}_n^2\right) \\ & =\frac{1}{n-1}\left(\sum x_i^2-2 n \frac{1}{n} \sum x_i \bar{x}_n+\sum \bar{x}_n^2\right) \\ & =\frac{1}{n-1}\left(\sum x_i^2-2 n \bar{x}_n^2+\sum \bar{x}_n^2\right) \\ & =\frac{1}{n-1} \left(\sum x_i^2-2 \sum \bar{x}_n^2+\sum \bar{x}_n^2\right) \\ & =\frac{1}{n-1}\left(\sum x_i^2-\sum \bar{x}_n^2\right) \\ & =\frac{n}{n-1}\left(\frac{1}{n} \sum x_i^2-\bar{x}_n^2\right) \stackrel{P}{\longrightarrow} 1 \cdot\left(E\left[x_1^2\right]-E\left[x_1\right]^2\right)=\sigma^2.\end{aligned}$

And thus by the definition of consistency $s^2$ is a consistent estimator of the variance.

$\endgroup$
6
  • $\begingroup$ As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Nov 23 at 1:11
  • $\begingroup$ @Community This is pretty clear to me (actually even included too many intermediate steps -- in a good way though). What is your criterion of judging this answer as "unclear"? $\endgroup$
    – Zhanxiong
    Nov 23 at 2:51
  • $\begingroup$ FYI, @Zhanxiong, I reviewed it as okay in the late answers queue. $\endgroup$ Nov 23 at 2:53
  • $\begingroup$ I might be wrong, but I'm not yet convinced that $\bar{x}_n^2 \rightarrow \mathbb{E}(X_1)^2$ (in probability) unless you impose finite kurtosis. Can you please elaborate on that step to show how you get this result? In particular, do you need to assume finite kurtosis for this part? (You need to assume finite variance for convergence of the sample mean, so I would think you need finite kurtosis for convergence of its square.) $\endgroup$
    – Ben
    Nov 23 at 2:59
  • $\begingroup$ @Zhanxiong: Yes, I suppose I am thinking about proofs of WLLN that use finite variance, but of course that is not a required condition if I recall correctly. $\endgroup$
    – Ben
    Nov 23 at 3:08
4
$\begingroup$

Vanishing variance (and resulting convergence in mean square) occurs if the underlying distribution has finite kurtosis

The other answer here considers the case of a sample variance of IID normally distributed random variables, but this is only one special case which rarely holds for most cases of interest. I will proceed for a more general case and look at convergence in mean-square. Your question is a bit ambiguous about what type of convergence you are interested in, but convergence in mean-square implies convergence in probability so this will hopefully be a useful result for you.

We can proceed for the more general case by considering IID random variables from any underlying distribution with variance $\sigma^2$ and kurtosis $\kappa$. Given $n$ sample values, the variance of the sample variance (see e.g., O'Neill 2014, p. 284) is:

$$\mathbb{V}(S_n^2) = \Big( \kappa - \frac{n-3}{n-1} \Big) \frac{\sigma^4}{n}.$$

(Please have a look at the linked paper if you would like to see the derivation of this result.) Now, if the kurtosis of the underlying distribuion is infinite ($\kappa = \infty$) then the variance of the sample variance is also infinite for all $n$ and you won't get convergence in mean-square for the sample variance. However, if the kurtosis is finite ($\kappa < \infty$) then we get:

$$\begin{align} \lim_{n \rightarrow \infty} \mathbb{V}(S_n^2) &= \lim_{n \rightarrow \infty} \Big( \kappa - \frac{n-3}{n-1} \Big) \frac{\sigma^4}{n} \\[6pt] &= (\kappa - 1) \sigma^4 \lim_{n \rightarrow \infty} \frac{1}{n} \\[12pt] &= 0. \\[6pt] \end{align}$$

Since the sample variance is an unbiased estimator of $\sigma^2$, this is sufficient to show that the sample variance converges in mean-square (and therefore also converges in probability) to $\sigma^2$. This means that the sample variance converges to the true variance for IID data, so long as the underlying distribution has finite kurtosis.

$\endgroup$
5
  • 1
    $\begingroup$ Like the accepted answer, your proposed solution also imposed conditions that are overly strong (i.e., assume finite fourth moments). The question is a fairly elementary result which only needs i.i.d. assumption + finite 2nd moments (as correctly detailed in @Julian Singh's answer). $\endgroup$
    – Zhanxiong
    Nov 23 at 2:47
  • 1
    $\begingroup$ Of course, you might be misled by the way of the original question was asked. After OP first asked the consistency of $S_n^2$, later he added "meaning that ..., $\operatorname{Var}(s^2)$ approaches 0 and it is unbiased". OP seems didn't understand the concept of consistency very well and mistakenly claimed the equivalence between the consistency and vanishing $\operatorname{var}(s^2)$. $\endgroup$
    – Zhanxiong
    Nov 23 at 3:06
  • $\begingroup$ @Zhanxiong: Yes, I was referring here to the vanishing variance and convergence in mean-square. I've edited the answer to make this more clear. $\endgroup$
    – Ben
    Nov 23 at 3:10
  • $\begingroup$ Thank you. Considering the question title is "How to prove $s^2$ is a consistent estimator of $\sigma^2$", it would be even better and more helpful if you can clarify that to have consistency we only need the 2nd finite moment condition and perhaps help OP understand the difference between consistency and converging in squared mean (or in fourth moments). $\endgroup$
    – Zhanxiong
    Nov 23 at 3:18
  • $\begingroup$ @Zhanxiong: I have edited the answer to make it clearer that I am focussing on the mean-square analysis. I have declined to go into further detail about the kurtosis condition because this comment section and the other answer do a good job of discussing this. $\endgroup$
    – Ben
    Nov 23 at 5:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.