Expected vallue calculation of i.i.d. random variables

Question

Suppose $X_1,X_2,\ldots,X_n$ are a sequence of i.i.d. random variables with mean $\mu$ and variance $\sigma^2$. Define the sample mean $\bar{X} := \frac{1}{n} \sum_{i=1}^{n} X_i$, which we know is an unbiased estimator of the sample mean with mean $\mu$ and variance $\sigma^2/n$, i.e.

\begin{align*} \mathbb{E}[\bar{X}] &= \mu, \\ \textrm{Var}(\bar{X}) := \mathbb{E}[(\bar{X} - \mu)^2] &= \frac{\sigma^2}{n}. \end{align*}

I am interested in calculating the expected value of the quantity $Z_n := \sum_{i=1}^{n} (X_i - \bar{X})^2$, but my results don't make sense. First, I expand the expectation to get

\begin{align*} \mathbb{E}[Z_n] &= \mathbb{E}\bigg[\sum_{i=1}^{n}(X_i - \bar{X})^2\bigg] = \mathbb{E}\bigg[(X_1 - \bar{X})^2 + \ldots + (X_n - \bar{X})^2\bigg] \\ &= \sum_{i=1}^{n} \mathbb{E}[(X_i - \bar{X})^2] = \sum_{i=1}^{n} \mathbb{E}[X_i^2 + \bar{X}^2 - 2X_i\bar{X}] \\ &= \sum_{i=1}^{n}(\mathbb{E}[X_i^2] + \mathbb{E}[\bar{X}^2] - 2\mathbb{E}[X_i\bar{X}]). \end{align*}

Thus, there are three expectations to compute. First, since each $X_i$ is i.i.d, it follows from the definition of variance that $\sigma^2 = \mathbb{E}[X_i^2] - \mathbb{E}[X_i]^2 \Rightarrow \mathbb{E}[X_i^2] = \sigma^2 + \mu^2$. Additionally, the same argument applies to the expected value of the squared sample mean, i.e., $\mathbb{E}[\bar{X}^2] = \sigma^2/n + \mu^2$.

The last expectation, $\mathbb{E}[X_i,\bar{X}]$ is a bit more tricky to compute. First, let us plug in what we currently have, which gives

$$ \mathbb{E}[Z_n] = \sum_{i=1}^{n} \bigg[(\sigma^2 + \mu^2) + \bigg(\frac{\sigma^2}{n} + \mu^2\bigg) -2\mathbb{E}[X_i\bar{X}]\bigg] = 2\mu^2n + (n+1)\sigma^2 - 2\sum_{i=1}^{n}\mathbb{E}[X_i\bar{X}]. $$

Now, for the last term, let us use the definition of the sample mean to get

$$ \sum_{i=1}^{n} \mathbb{E}[X_i\bar{X}] = \sum_{i=1}^{n} \mathbb{E}\bigg[X_i\bigg(\frac{1}{n}\sum_{j=1}^{n}X_j\bigg)\bigg] = \frac{1}{n}\sum_{i=1}^{n}\sum_{j=1}^{n}\mathbb{E}[X_iX_j], $$ where I used the linearity of the expectation in the last equality. Noting that $\textrm{Cov}(X_i,X_j) = 0$ for all $i \neq j$ since each $X_i$ are independent, we see that $\textrm{Cov}(X_i,X_j) = \mathbb{E}[X_iX_j] - \mu^2 = 0$ for all $i \neq j$, which implies $\mathbb{E}[X_iX_j] = \mu^2$ for all $i \neq j$. Similarly, for all $i = j$, we have $\textrm{Cov}(X_i,X_j) = \textrm{Cov}(X_i,X_i) = \sigma^2$, by definition. Thus, if we break up that double sum into a double sum when $i = j$ and a double sum when $i \neq j$, we get

$$ \sum_{i=1}^{n}\mathbb{E}[X_i\bar{X}] = \frac{1}{n}(n\mu^2 + n\sigma^2) = \mu^2 + \sigma^2. $$

Plugging this back in gives

$$ \mathbb{E}[Z_n] = 2\mu^2n + (n + 1)\sigma^2 - 2(\sigma^2 + \mu^2) = \boxed{ (n-1)(2\mu^2 + \sigma^2) } $$

My question is what is the physical significance of this $Z_n$ that I'm trying to calculate, and is the calculation correct?

Answer 1

Let's start with: \begin{align*} Z_n&=\sum_{i=1}^n(X_i-\bar{X})^2=\sum_{i=1}^n(X_i^2-2X_i\bar{X}+\bar{X}^2)\\ &=\sum_{i=1}^nX_i^2-2\left(\sum_{i=1}^nX_i\right)\bar{X}+n\bar{X}^2\\ &=\sum_{i=1}^nX_i^2-2n\bar{X}\bar{X}+n\bar{X}^2=\sum_{i=1}^nX_i^2-n\bar{X}^2 \end{align*} Then $$E[Z_n]=E\left[\sum_{i=1}^nX_i^2\right]-nE[\bar{X}^2]\overset{\mathrm{iid}}{=}nE[X^2]-nE[\bar{X}^2]$$ where $X\sim X_i$, $i=1,\dots,n$. Since $\sigma^2=E[X^2]-\mu^2$ and $[\bar{X}^2]=\frac{\sigma^2}{n}+\mu^2$, $$E[Z_n]=n\left(\sigma^2+\mu^2-\frac{\sigma^2}{n}+\mu^2\right)=(n-1)\sigma^2$$ But what is $Z_n$? $Z_n$ is just $$Z_n=n\hat\sigma^2_n=(n-1)S^2_n$$ where $\hat\sigma^2_n=\frac1n\sum_{i=1}^n(X_i-\bar{X})^2$ is the sample variance and $S^2_n=\frac{1}{n-1}\sum_{i=1}^n(X_i-\bar{X})^2$ is the unbiased sample variance: $$E[\hat\sigma^2_n]=\frac{n-1}{n}\sigma^2,\qquad E[S^2_n]=\sigma^2$$