Suppose $X_1,X_2,\ldots,X_n$ are a sequence of i.i.d. random variables with mean $\mu$ and variance $\sigma^2$. Define the sample mean $\bar{X} := \frac{1}{n} \sum_{i=1}^{n} X_i$, which we know is an unbiased estimator of the sample mean with mean $\mu$ and variance $\sigma^2/n$, i.e.

\begin{align*} \mathbb{E}[\bar{X}] &= \mu, \\ \textrm{Var}(\bar{X}) := \mathbb{E}[(\bar{X} - \mu)^2] &= \frac{\sigma^2}{n}. \end{align*}

I am interested in calculating the expected value of the quantity $Z_n := \sum_{i=1}^{n} (X_i - \bar{X})^2$, but my results don't make sense. First, I expand the expectation to get

\begin{align*} \mathbb{E}[Z_n] &= \mathbb{E}\bigg[\sum_{i=1}^{n}(X_i - \bar{X})^2\bigg] = \mathbb{E}\bigg[(X_1 - \bar{X})^2 + \ldots + (X_n - \bar{X})^2\bigg] \\ &= \sum_{i=1}^{n} \mathbb{E}[(X_i - \bar{X})^2] = \sum_{i=1}^{n} \mathbb{E}[X_i^2 + \bar{X}^2 - 2X_i\bar{X}] \\ &= \sum_{i=1}^{n}(\mathbb{E}[X_i^2] + \mathbb{E}[\bar{X}^2] - 2\mathbb{E}[X_i\bar{X}]). \end{align*}

Thus, there are three expectations to compute. First, since each $X_i$ is i.i.d, it follows from the definition of variance that $\sigma^2 = \mathbb{E}[X_i^2] - \mathbb{E}[X_i]^2 \Rightarrow \mathbb{E}[X_i^2] = \sigma^2 + \mu^2$. Additionally, the same argument applies to the expected value of the squared sample mean, i.e., $\mathbb{E}[\bar{X}^2] = \sigma^2/n + \mu^2$.

The last expectation, $\mathbb{E}[X_i,\bar{X}]$ is a bit more tricky to compute. First, let us plug in what we currently have, which gives

$$ \mathbb{E}[Z_n] = \sum_{i=1}^{n} \bigg[(\sigma^2 + \mu^2) + \bigg(\frac{\sigma^2}{n} + \mu^2\bigg) -2\mathbb{E}[X_i\bar{X}]\bigg] = 2\mu^2n + (n+1)\sigma^2 - 2\sum_{i=1}^{n}\mathbb{E}[X_i\bar{X}]. $$

Now, for the last term, let us use the definition of the sample mean to get

$$ \sum_{i=1}^{n} \mathbb{E}[X_i\bar{X}] = \sum_{i=1}^{n} \mathbb{E}\bigg[X_i\bigg(\frac{1}{n}\sum_{j=1}^{n}X_j\bigg)\bigg] = \frac{1}{n}\sum_{i=1}^{n}\sum_{j=1}^{n}\mathbb{E}[X_iX_j], $$ where I used the linearity of the expectation in the last equality. Noting that $\textrm{Cov}(X_i,X_j) = 0$ for all $i \neq j$ since each $X_i$ are independent, we see that $\textrm{Cov}(X_i,X_j) = \mathbb{E}[X_iX_j] - \mu^2 = 0$ for all $i \neq j$, which implies $\mathbb{E}[X_iX_j] = \mu^2$ for all $i \neq j$. Similarly, for all $i = j$, we have $\textrm{Cov}(X_i,X_j) = \textrm{Cov}(X_i,X_i) = \sigma^2$, by definition. Thus, if we break up that double sum into a double sum when $i = j$ and a double sum when $i \neq j$, we get

$$ \sum_{i=1}^{n}\mathbb{E}[X_i\bar{X}] = \frac{1}{n}(n\mu^2 + n\sigma^2) = \mu^2 + \sigma^2. $$

Plugging this back in gives

$$ \mathbb{E}[Z_n] = 2\mu^2n + (n + 1)\sigma^2 - 2(\sigma^2 + \mu^2) = \boxed{ (n-1)(2\mu^2 + \sigma^2) } $$

My question is what is the physical significance of this $Z_n$ that I'm trying to calculate, and is the calculation correct?


Let's start with: \begin{align*} Z_n&=\sum_{i=1}^n(X_i-\bar{X})^2=\sum_{i=1}^n(X_i^2-2X_i\bar{X}+\bar{X}^2)\\ &=\sum_{i=1}^nX_i^2-2\left(\sum_{i=1}^nX_i\right)\bar{X}+n\bar{X}^2\\ &=\sum_{i=1}^nX_i^2-2n\bar{X}\bar{X}+n\bar{X}^2=\sum_{i=1}^nX_i^2-n\bar{X}^2 \end{align*} Then $$E[Z_n]=E\left[\sum_{i=1}^nX_i^2\right]-nE[\bar{X}^2]\overset{\mathrm{iid}}{=}nE[X^2]-nE[\bar{X}^2]$$ where $X\sim X_i$, $i=1,\dots,n$. Since $\sigma^2=E[X^2]-\mu^2$ and $[\bar{X}^2]=\frac{\sigma^2}{n}+\mu^2$, $$E[Z_n]=n\left(\sigma^2+\mu^2-\frac{\sigma^2}{n}+\mu^2\right)=(n-1)\sigma^2$$ But what is $Z_n$? $Z_n$ is just $$Z_n=n\hat\sigma^2_n=(n-1)S^2_n$$ where $\hat\sigma^2_n=\frac1n\sum_{i=1}^n(X_i-\bar{X})^2$ is the sample variance and $S^2_n=\frac{1}{n-1}\sum_{i=1}^n(X_i-\bar{X})^2$ is the unbiased sample variance: $$E[\hat\sigma^2_n]=\frac{n-1}{n}\sigma^2,\qquad E[S^2_n]=\sigma^2$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.