Consider $\{X_i\}$ i.i.d. sample of multivariate (in $\mathbb{R}^J$) with mean $\mu$ and variance $\Sigma$. I've been asked to show that $\hat{\Sigma}\equiv\frac{1}{n}\sum_{i=1}^n (X_i-\bar{X}_n) (X_i-\bar{X}_n)'$ converges almost surely to $\Sigma$.

I have no idea how to star that, besides using Central Limit Theorem on $\bar{X}_n$, and I don't know what to do with that. Can anyone give me some hints and ideas on how to proceed? Thank you for your help.

  • 2
    $\begingroup$ The CLT, by itself, even when the random variables are defined on the same probability space, would not yield the necessary conclusion anyway. $\endgroup$
    – cardinal
    Oct 6, 2015 at 23:47
  • 2
    $\begingroup$ Have you heard of the (Strong) Law of Large numbers? $\endgroup$
    – JohnK
    Oct 7, 2015 at 0:11
  • $\begingroup$ Yes I have. I think I will get the result if I apply it to $Z_i \equiv (X_i-\bar{X}_n)(X_i-\bar{X}_n)'$ and show that $E(Z_i)=\Sigma$ and argue that SLLN is valid for matrices (I only have seen the result for vectors). Thanks for the tips! Any other comments? $\endgroup$ Oct 7, 2015 at 2:53
  • $\begingroup$ The SLLN thing for matrices might be doable by stacking the columns of the matrix in a really big vector, but I'm struggling with the $E(Z_i)=\Sigma$. $\endgroup$ Oct 7, 2015 at 3:17

1 Answer 1


The idea is to open the sample variance: $$ \hat{\Sigma}=\frac{1}{n}\sum_{i=1}^nX_iX_i'-\bar{X}_n\bar{X}_n' $$ and notice that by the Strong Law of Large Numbers and the Continuous Mapping Theorem (actually a corollary of it, probably known as "Algebra of Stochastic Convergence"): $$ \bar{X}_n\overset{a.s.}{\to}\mu\implies\bar{X}_n\bar{X}_n'\overset{a.s.}{\to}\mu\mu'\\ \frac{1}{n}\sum_{i=1}^nX_iX_i'\overset{a.s.}{\to}E(X_iX_i') $$ where the convergence on the second line happens because the variance exists, so that the expectation of the product $X_iX_i'$ is well-defined. And using Continuous Mapping theorem again we can subtract both and maintain the convergence: $$ \hat{\Sigma}\overset{a.s.}{\to}\Sigma $$

  • $\begingroup$ How do you prove, in the the third matheatical (symbolic), line, that the random matrices $X_i X_i'$ satisfy the conditions for the strong law of large numbers to begin with? That is, how do you show that $X_iX_i'$ are iid, starting from the assumption that $X_i$ are iid? $\endgroup$
    – Mathmath
    Oct 24, 2019 at 13:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.