# Almost sure convergence of the minimum eigenvalue of a sample covariance matrix

I was wondering if someone could provide a reference to the following result. Consider the $$p\times p$$ sample matrix $$\frac{1}{n} \sum_{i=1}^n x_i x_i',$$ where $$x_i$$ are i.i.d. $$p\times 1$$ random vectors (with all the finite moments), we can assume they have mean zero. I conjecture that the minimum eigenvalue of this sample matrix converges almost surely to the minimum eigenvalue of the population matrix $$E[x_i x_i']$$.

• Consider $\sum_{i=1}^n x_i x_i^Tv$ where v is the eigenvector of the minimal eigenvalue of$E[x_1 x_1^T$ Mar 16, 2020 at 20:44
• @Sebastian I understand that we will have convergence a.s. of $\frac{1}{n}\sum_{i=1}^n x_i x_i^T v$ to $\lambda_{min}v$. But in this case how do I relate $\sum_{i=1}^n x_i x_i^T v$ to the minimum eigenvalue of $\sum_{i=1}^n x_i x_i^T$ given that it can be different from $\lambda_{min}$ and the eigenvector for the sample matrix can be different as well? Mar 17, 2020 at 8:24
• It also converges against $E[x_1x_1^T]v$ Mar 17, 2020 at 21:41

Let $$\Sigma = \mathbb{E}[x_1 x_1^T]$$. By the law of large numbers, if we hold the dimension $$p$$ fixed and let $$n \to \infty$$,
$$\frac{1}{n} \sum_{i=1}^n x_i x_i^T \stackrel{\mathrm{a.s.}}{\to} \Sigma = \mathbb{E}[x_1 x_1^T].$$
If we assume the (reasonable) condition that $$\Sigma$$ has distinct eigenvalues and eigenvectors, then the eigenvalues are a continuous function of $$\Sigma$$. See The Matrix Cookbook (Petersen and Pedersen), which is linked here) for a reference of this fact. As a result, by the continuous mapping theorem, we have that $$\lambda_{\min}\left(\frac{1}{n} \sum_{i=1}^n x_i x_i^T\right) \stackrel{\mathrm{a.s.}}{\to} \lambda_{\min}(\Sigma).$$
A final caveat which is related to your question: this result only holds asymptotically when $$n$$ is many times larger than $$p$$. More "modern" theory often considers an asymptotic regime where $$n, p$$ both grow to infinity at a linear rate, meaning $$\lim_{n \to \infty} \frac{p}{n} = \gamma \in (0, \infty)$$. In this case, the minimum eigenvalue of the sample covariance matrix does not usually converge to the minimum eigenvalue of the covariance matrix: usually, it is inconsistent (too small asymptotically). See The Smallest Eigenvalue of a Large Dimensional Wishart Matrix by JW Silverstein as a canonical reference, although there has been a lot of additional progress in this field in the past 20 years.