Intuitive explanation for Marchenko-Pastur law

Question

I am looking for an intuitive reasoning behind the Marchenko Pastur law, which is described as a law of large numbers analog for random matrices. I know the law gives the probability density function of eigenvalues values of a large Wishart matrix whose dimensions tend to infinity, but I am curious of the corollaries for covariance matrices of large dimensional $n\times p$ matrices $X$ with correlation between columns.

Let $\gamma=p/n$. How does MP law provide the lower $\lambda^-$ and upper bound $\lambda^+$ of the eigenvalues for $\mathbf{C}$ derived from an $n \times p$ Wishart matrix, without knowing the variance of its random variates? Secondly, given $n$ and $p$, why is it that the eigenvalues of ${E(X^TX)}$ which exceed $\lambda^+$ are considered to be the signal eigenvalues but anything below are considered noise? What do eigenvalues to the left of the interval $(\lambda^-, \lambda^+)$ indicate?

Answer 1

You are asking several different questions here. you are asking 1) what is the marchenko-pastur law when there is some correlation between the columns of the Wishart matrix, 2) how do we find $\lambda^+, \lambda^-$ when $\sigma$ is not known, 3) why are the eigenvalues above these thresholds considered signal, and 4) what do eigenvalues below that value signify?

I will give a cursory answer to each one, but please consider cutting it down to a single question, or posting separate questions.

1. It is no longer a standard Wishart matrix if there is correlation in the variates. A standard wishart matrix has i.i.d. gaussian distributed entries. What you're referring to is a "structured" random matrix. I don't believe there is a known solution for arbitrary correlation structure. But there are certain special cases that are known, and they are not as clean and simple as the Marchenko-Pastur law. For a primer on structured random matrices, see this article by Van Handel.

2. I'm not sure what you're getting at -- the marchenko pastur law assumes you know the variance of the i.i.d. gaussian distributed entries. If you don't have that, you don't know the distribution. Perhaps you meant without knowing the covariance, i.e. the correlation structure. Based on your first question, I'll assume this is what you're asking. Again, the answer is that the Marchenko pastur law assumes no covariance -- i.e. the true covariance matrix is a constant multiple of the identity matrix, and the sample covariance matrix is some constant multiple of the identity + random noise. If you are assuming some true covariance, it is no longer described by Marchenko-Pastur.

3. The reason the eigenvalues above this threshold are considered signal is that they indicate that the true covariance matrix is not in fact diagonal. Let's assume for simplicity for a second that $\sigma=1$. Then the idea behind using it to "de-noise" a covariance matrix is something like the following: we know that even if we assume 0 true correlation (or a covariance matrix equal to the identity), due to random chance there is going to be some observed correlation in real-world matrices unless p much larger than N. If the sample correlation were exactly the identity, all eigenvalues would be 1, with eigenvectors equal to the standard basis vectors. Instead, this noise creates some distribution around $\lambda=1$, which interestingly does not vanish even as $N,p \to \infty$, as long as $\frac{N}{p}$ is constant. Now, if there is some true correlation structure, this would imply that there are some eigenvalues "significantly" higher than 1. For example, suppose there are two variates $X_1,X_2$ that are actually highly correlated. Then (in expectation at least) there will be an eigenportfolio with equal weight on these two variates (weight = $\frac{1}{\sqrt{2}}$) and 0 weight everywhere else, such that the variance of this portfolio is $Var(\frac{1}{\sqrt{2}}X_1)+Var(\frac{1}{\sqrt{2}}X_2)+2Cov(\frac{1}{\sqrt{2}}X_1,\frac{1}{\sqrt{2}}X_2)$. The first two terms sum to 1, the second term signifies a "significant increase" above 1. Thus, anything that is significantly higher than 1 by a margin greater than what you would see by random chance (above $\lambda^+$) signifies "real" correlation structure.

You may be getting confused because we are using the assumption of no correlation in order to find the true correlation matrix. But if you think about it, is this very different from assuming any other kind of null hypothesis in order to show that the alternative hypothesis is true? It should be noted here that the mathematical framework used to determine which eigenvalues represent true correlation, which is based on the uncorrelated null hypothesis, is completely separate from the process of actually "de-noising" the corelation matrix, i.e. "finding the true correlation", i.e. performing PCA on the matrix, which absolutely does not continue this assumption of no correlation. The only connection is that, perhaps somewhat paradoxically, you use the former in order to select the number of principal components to use for the latter.

4. You can think of extremal low eigenvalues as being a natural counterpoint to any extremal high eigenvalues. Note that following the example above, the existence of that high correlation implies that in order for each variate to have an observed variance of 1, the other eigenportolfio including those two variates, which would have them with opposite signs in order to be orthogonal to the first one, would necessarily have a much smaller variance than 1, since $Cov(aX_1,-bX_2)<0$ for a,b>0. Let's take it to the extreme: if they were 100% correlated, this would imply near collinearity of the sample correlation matrix, i.e. it has an eigenvalue of nearly 0 corresponding to the eigenportfolio with $a=b=\frac{1}{\sqrt{2}}$. In practice, if you see lots of extremely high eigenvalues of the correlation, you will also find that the bulk distribution will be shifted slightly to the left of what is predicted by Marchenko-Pastur, to "compensate" for this.