Suppose $X$ follows some multivariate distribution with zero mean and Identity covariance matrix. Suppose $X$ is N dimensional. Suppose $R$ is the sample correlation matrix, calculated based on n observations where N>>n. What is the expected value of the largest eigen value of $R$?
3
-
1$\begingroup$ If you are ok with empirical distributions, it may be simple enough to sample those $X$'s compute $R$ and extract the eigenvalues, to get the empirical distribution. $\endgroup$– CryoCommented Jan 17 at 20:57
-
$\begingroup$ @Cryo I am looking for a theoretical solution. I ran simulation and am getting slightly greater 1. $\endgroup$– debCommented Jan 18 at 1:02
-
1$\begingroup$ This is a matter of random matrix theory. There are limiting theorems. See The Eigenvalues of Random Matrices. There are expressions for density of eigenvalues. $\endgroup$– patagonicusCommented Jan 20 at 7:48
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
It depends on the correlation between the features (dimensions), i.e., the data. Firstly, there will be some zero eigenvalues because when $p \gg n$, $\mathbf{R}$ will be positive semi-definite ($\lambda_j \geq 0$), whereas when $p<n$, $\mathbf{R}$ will be positive definite ($\lambda_j > 0$). Secondly, even though there will be some zero eigenvalues, the remaining eigenvalues will pick up the slack for whatever correlation remains. The values of the remaining non-zero eigenvalues will depend on the off-diagonal values in $\mathbf{R}$.
Another rule for eigenvalues when using $\mathbf{R}$ (vs. the sample covariance matrix $\mathbf{S}$) is that the sum of eigenvalues will be equal to the number of dimensions $p$, that is $\sum_{j=1}^p \lambda_j = p$. (It can be proven that the sum of the eigenvalues of a square matrix equals the trace of the matrix, and in this case the trace is $p$ because a correlation matrix has all 1s on the main diagonal.) However, the only rule-of-thumb for your case is that there will be some zero eigenvalues.
Almost anything regarding eigendecomposition can be determined using an algorithm. Correlation and covariance matrices are in the family of "real-square symmetric" matrices. There are hundreds of algorithms for eigenvalues/eigenvectors of a real-square symmetric matrix. The analytic problem is called the "symmetric eigenvalue problem", so your search should be e.g. "largest eigenvalue symmetric matrix", which will give you something like this
-
$\begingroup$ Perhaps you could expand on this answer to explain why the sum of the eigenvalues is $p$. $\endgroup$– Sycorax ♦Commented Jan 17 at 18:23
-
$\begingroup$ Sure, see modified answer - it's because, by definition, the sum of $\lambda_j$ is equal to the trace of $\mathbf{R}$, or the sum of all the diagonal elements, each of which are $1$. This is not true for the covariance matrix, since it's diagonal elements are the variance of each feature, $\sigma^2(Y_j)$. $\endgroup$– wjktrsCommented Jan 17 at 18:37
-
$\begingroup$ Thanks @wjktrs. That's all true. But my main question is still open. Given all you said, what can we say about the expected value of $\lambda_{max}$? Or for that matter $\frac{\lambda_{max}}{\Sigma_i \lambda_i}=\frac{\lambda_{max}}{n}$ or $\frac{\lambda_{max}}{\Sigma_i \lambda_i}=\frac{\lambda_{max}}{p}$ in your set-up? I ran some simulation. I am getting slightly greater than $\frac{1}{p}$. I am looking for an theoretical expected value. $\endgroup$– debCommented Jan 18 at 1:00