1
$\begingroup$

I was wondering if someone could provide a reference to the following result. Consider the $p\times p $ sample matrix $$\frac{1}{n} \sum_{i=1}^n x_i x_i',$$ where $x_i$ are i.i.d. $p\times 1$ random vectors (with all the finite moments), we can assume they have mean zero. I conjecture that the minimum eigenvalue of this sample matrix converges almost surely to the minimum eigenvalue of the population matrix $E[x_i x_i']$.

$\endgroup$
3
  • 3
    $\begingroup$ Consider $\sum_{i=1}^n x_i x_i^Tv$ where v is the eigenvector of the minimal eigenvalue of$E[x_1 x_1^T$ $\endgroup$
    – Sebastian
    Mar 16, 2020 at 20:44
  • $\begingroup$ @Sebastian I understand that we will have convergence a.s. of $\frac{1}{n}\sum_{i=1}^n x_i x_i^T v$ to $\lambda_{min}v$. But in this case how do I relate $\sum_{i=1}^n x_i x_i^T v$ to the minimum eigenvalue of $\sum_{i=1}^n x_i x_i^T$ given that it can be different from $\lambda_{min}$ and the eigenvector for the sample matrix can be different as well? $\endgroup$
    – Neznajka
    Mar 17, 2020 at 8:24
  • $\begingroup$ It also converges against $E[x_1x_1^T]v$ $\endgroup$
    – Sebastian
    Mar 17, 2020 at 21:41

1 Answer 1

3
$\begingroup$

Let $\Sigma = \mathbb{E}[x_1 x_1^T]$. By the law of large numbers, if we hold the dimension $p$ fixed and let $n \to \infty$,

$$\frac{1}{n} \sum_{i=1}^n x_i x_i^T \stackrel{\mathrm{a.s.}}{\to} \Sigma = \mathbb{E}[x_1 x_1^T]. $$

If we assume the (reasonable) condition that $\Sigma$ has distinct eigenvalues and eigenvectors, then the eigenvalues are a continuous function of $\Sigma$. See The Matrix Cookbook (Petersen and Pedersen), which is linked here) for a reference of this fact. As a result, by the continuous mapping theorem, we have that $$\lambda_{\min}\left(\frac{1}{n} \sum_{i=1}^n x_i x_i^T\right) \stackrel{\mathrm{a.s.}}{\to} \lambda_{\min}(\Sigma). $$

A final caveat which is related to your question: this result only holds asymptotically when $n$ is many times larger than $p$. More "modern" theory often considers an asymptotic regime where $n, p$ both grow to infinity at a linear rate, meaning $\lim_{n \to \infty} \frac{p}{n} = \gamma \in (0, \infty)$. In this case, the minimum eigenvalue of the sample covariance matrix does not usually converge to the minimum eigenvalue of the covariance matrix: usually, it is inconsistent (too small asymptotically). See The Smallest Eigenvalue of a Large Dimensional Wishart Matrix by JW Silverstein as a canonical reference, although there has been a lot of additional progress in this field in the past 20 years.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.