# Almost sure convergence of sample variance with iid sample

Consider $\{X_i\}$ i.i.d. sample of multivariate (in $\mathbb{R}^J$) with mean $\mu$ and variance $\Sigma$. I've been asked to show that $\hat{\Sigma}\equiv\frac{1}{n}\sum_{i=1}^n (X_i-\bar{X}_n) (X_i-\bar{X}_n)'$ converges almost surely to $\Sigma$.

I have no idea how to star that, besides using Central Limit Theorem on $\bar{X}_n$, and I don't know what to do with that. Can anyone give me some hints and ideas on how to proceed? Thank you for your help.

• The CLT, by itself, even when the random variables are defined on the same probability space, would not yield the necessary conclusion anyway. Oct 6, 2015 at 23:47
• Have you heard of the (Strong) Law of Large numbers? Oct 7, 2015 at 0:11
• Yes I have. I think I will get the result if I apply it to $Z_i \equiv (X_i-\bar{X}_n)(X_i-\bar{X}_n)'$ and show that $E(Z_i)=\Sigma$ and argue that SLLN is valid for matrices (I only have seen the result for vectors). Thanks for the tips! Any other comments? Oct 7, 2015 at 2:53
• The SLLN thing for matrices might be doable by stacking the columns of the matrix in a really big vector, but I'm struggling with the $E(Z_i)=\Sigma$. Oct 7, 2015 at 3:17

The idea is to open the sample variance: $$\hat{\Sigma}=\frac{1}{n}\sum_{i=1}^nX_iX_i'-\bar{X}_n\bar{X}_n'$$ and notice that by the Strong Law of Large Numbers and the Continuous Mapping Theorem (actually a corollary of it, probably known as "Algebra of Stochastic Convergence"): $$\bar{X}_n\overset{a.s.}{\to}\mu\implies\bar{X}_n\bar{X}_n'\overset{a.s.}{\to}\mu\mu'\\ \frac{1}{n}\sum_{i=1}^nX_iX_i'\overset{a.s.}{\to}E(X_iX_i')$$ where the convergence on the second line happens because the variance exists, so that the expectation of the product $X_iX_i'$ is well-defined. And using Continuous Mapping theorem again we can subtract both and maintain the convergence: $$\hat{\Sigma}\overset{a.s.}{\to}\Sigma$$
• How do you prove, in the the third matheatical (symbolic), line, that the random matrices $X_i X_i'$ satisfy the conditions for the strong law of large numbers to begin with? That is, how do you show that $X_iX_i'$ are iid, starting from the assumption that $X_i$ are iid? Oct 24, 2019 at 13:18