Efficient proof of Bessel's correction

Question

I am beginning to learn the fundamentals of statistics, and I am reading the proof of why Bessel's correction in the estimate of the variance works from the wikipedia page about it. I understand everything, except the following dramatic simplification of calculation (reproduced here):

... here we have (by independence, symmetric cancellation and equal distribution) $$... \mathbb{E} \Big[ \displaystyle\sum_{j=1}^n\sum_{l=1}^n (x_k - x_j)(x_k - x_l) \Big] = n(n-1)\mathbb{E}[X_1^2] - n(n-1)\mathbb{E}[X_1]^2$$

and I don't see how the calculation follows so easily. What do the author of the page mean precisely by "by independence, symmetric cancellation and equal distribution"?

Answer 1

To begin with, note that \begin{align*} \sum_{j = 1}^n (x_k - x_j) = \sum_{l = 1}^n (x_k - x_l) = nx_k - n\bar{x}, \end{align*} whence \begin{align*} \sum_{j = 1}^n\sum_{l = 1}^n (x_k - x_j)(x_k - x_l) = (nx_k - n\bar{x})^2.\tag{1}\label{1} \end{align*} Now the result easily follows from the linearity of the expectation operator and the i.i.d. assumption of the sample $\{x_1, \ldots, x_n\}$ (some basic operations during the calculation should be self-explanatory): \begin{align*} & E[(nx_k - n\bar{x})^2] \\ =& n^2E[x_k^2] - 2n^2E[x_k\bar{x}] + n^2E[\bar{x}^2] \\ =& n^2E[x_1^2] - 2n^2(n - 1)\frac{1}{n}E[x_1]^2 - 2n^2\frac{1}{n}E[x_k^2] + n^2\operatorname{Var}(\bar{x}) + n^2(E[\bar{x}])^2 \\ =& n^2E[x_1^2] - 2n(n - 1)E[x_1]^2 - 2nE[x_1^2] + n\operatorname{Var}(x_1) + n^2E[x_1]^2 \\ =& n^2E[x_1^2] - 2n(n - 1)E[x_1]^2 - 2nE[x_1^2] + n(E[x_1^2] - E[x_1]^2) + n^2E[x_1]^2 \\ =& n(n - 1)E[x_1^2] - n(n - 1)E[x_1]^2. \end{align*}

---

Alternatively, if you are familiar with matrix operations, the proof can be made shorter and cleaner. To this end, with $x = \begin{bmatrix} x_1 & x_2 & \cdots & x_n\end{bmatrix}^\top$, $e$ being the $n$-by-$1$ column vector of all ones and $e_k$ being the $n$-by-$1$ vector of all zeroes but the $k$-th entry $1$, $\eqref{1}$ can be rewritten as \begin{align*} n^2(x_k - \bar{x})^2 = n^2(e_k^\top x - n^{-1}e^\top x)^2 = n^2x^\top (e_k - n^{-1}e)(e_k - n^{-1}e)^\top x := n^2x^\top \Lambda x. \end{align*} Now it follows by the quadratic form expectation formula that \begin{align*} & E[n^2(x_k - \bar{x})^2] = n^2E[x^\top \Lambda x] = n^2\operatorname{tr}(\Lambda\Sigma) + n^2\mu^\top\Lambda\mu. \tag{2}\label{2} \end{align*} By the i.i.d assumption, $\Sigma = \sigma^2I_{(n)}$, where $\sigma^2 = E[x_1^2] - E[x_1]^2$, and $\mu = E[x_1]e$. Therefore, \begin{align*} & \operatorname{tr}(\Lambda\Sigma) = \sigma^2\operatorname{tr}(\Lambda) \\ =& \sigma^2\operatorname{tr}((e_k - n^{-1}e)(e_k - n^{-1}e)^\top) \\ =& \sigma^2(e_k - n^{-1}e)^\top(e_k - n^{-1}e) \\ =& \sigma^2 (1 - 2n^{-1} + n^{-1}) = (1 - n^{-1})\sigma^2. \tag{3}\label{3} \\ & \mu^\top \Lambda\mu = E[x_1]^2e^\top(e_k - n^{-1}e)(e_k - n^{-1}e)^\top e = E[x_1]^2(1 - 1)^2 = 0. \tag{4}\label{4} \end{align*} Substituting $\eqref{3}$ and $\eqref{4}$ into $\eqref{2}$ yields \begin{align*} & E[n^2(x_k - \bar{x})^2] = n^2(1 - n^{-1})\sigma^2 = n(n - 1)(E[x_1^2] - E[x_1]^2), \end{align*} as desired.

Answer 2

I like to think about all the $j$s and $l$s as arranged into a big square. The top left element is where both indexes are $1$. The bottom right element is where both elements are $n$. It's kind of like a matrix with $n\times n$ elements. Along the diagonal the row equals the column index. Everywhere off the diagonal the indexes differ.

Then you also have $k$, which is fixed.

One trick is to break up the sum into a few unique terms, and then count the number of times those unique terms occur.

Again, are $n$ situations where $j=l$. When this happens, $$ \mathbb{E} \Big[ \displaystyle(x_k - x_j)(x_k - x_l) \Big] = \mathbb{E} \Big[ \displaystyle(x_k - x_l)^2 \Big] $$

Then there are are $n^2 - n$ (area of the square minus the number of diagonal elements) elements where $j\neq l$. You get to exploit independence a lot here.

$$ \mathbb{E} \Big[ \displaystyle (x_k - x_j)(x_k - x_l) \Big] = \mathbb{E} \Big[ \displaystyle x_k^2 -x_kx_l - x_j x_k +x_jx_l \Big] $$

The rest is linearity, identicalness, independence and alternative definitions or properties of variance.