10
$\begingroup$

I'm talking here about matrices of Pearson correlations.

I've often heard it said that all correlation matrices must be positive semidefinite. My understanding is that positive definite matrices must have eigenvalues $> 0$, while positive semidefinite matrices must have eigenvalues $\ge 0$. This makes me think that my question can be rephrased as "Is it possible for correlation matrices to have an eigenvalue $= 0$?"

Is it possible for a correlation matrix (generated from empirical data, with no missing data) to have an eigenvalue $= 0$, or an eigenvalue $< 0$? What if it was a population correlation matrix instead?

I read at the top answer to this question about covariance matrices that

Consider three variables, $X$, $Y$ and $Z = X+Y$. Their covariance matrix, $M$, is not positive definite, since there's a vector $z$ ($= (1, 1, -1)'$) for which $z'Mz$ is not positive.

However, if instead of a covariance matrix I do those calculations on a correlation matrix then $z'Mz$ comes out as positive. Thus I think that maybe the situation is different for correlation and covariance matrices.

My reason for asking is that I got asked over on stackoverflow, in relation to a question I asked there.

$\endgroup$
  • $\begingroup$ If, for example, two attributes are one thing, only having different names, the matrix is singular. If two attributes add to a constant, it is again singular, et cetera. $\endgroup$ – ttnphns Nov 21 '15 at 16:54
  • $\begingroup$ If a covariance matrix is singular correlation matrix is singular as well. $\endgroup$ – ttnphns Nov 21 '15 at 17:05
  • 2
    $\begingroup$ Near-duplicates: Is every correlation matrix positive semi-definite? which has less focus on the definite versus semi-definite angle, and Is every covariance matrix positive definite? which is relevant because a covariance is essentially a rescaled correlation. $\endgroup$ – Silverfish Nov 21 '15 at 21:20
15
$\begingroup$

Correlation matrices need not be positive definite.

Consider a scalar random variable X having non-zero variance. Then the correlation matrix of X with itself is the matrix of all ones, which is positive semi-definite, but not positive definite.

As for sample correlation, consider sample data for the above, having first observation 1 and 1, and second observation 2 and 2. This results in sample correlation being the matrix of all ones, so not positive definite.

A sample correlation matrix, if computed in exact arithmetic (i.e., with no roundoff error) can not have negative eigenvalues.

$\endgroup$
  • 4
    $\begingroup$ May be worth mentioning the possible effects of missing values on the sample correlation matrix. Numerical fuzz isn't the only reason to get a negative eigenvalue in a sample correlation/covariance matrix. $\endgroup$ – Silverfish Nov 21 '15 at 21:22
  • 1
    $\begingroup$ Yes, I didn't make it explicit, but I was assuming, per the question statement, "with no missing data". Once you get into the wild, wacky world of missing data and adjustments therefor, anything goes. $\endgroup$ – Mark L. Stone Nov 21 '15 at 21:35
  • $\begingroup$ Yes, sorry, you're quite right the question said "no missing data" - just thought it worth mentioning somewhere since future searchers might be interested even if the OP's appetite is sated! $\endgroup$ – Silverfish Nov 21 '15 at 21:40
6
$\begingroup$

The answers by @yoki and @MarkLStone (+1 to both) both point out that a population correlation matrix can have zero eigenvalues if variables are linearly related (such as e.g. $X_1 = X_2$ in the example of @MarkLStone and $X_1 = 2X_2$ in the example of @yoki).

In addition to that, a sample correlation matrix will necessarily have zero eigenvalues if $n<p$, i.e. if the sample size is smaller than the number of variables. In this case covariance and correlation matrices will both be at most of rank $n-1$, so there will be at least $p-n+1$ zero eigenvalues. See Why is a sample covariance matrix singular when sample size is less than number of variables? and Why is the rank of covariance matrix at most $n-1$?

$\endgroup$
  • $\begingroup$ True 'dat. I suppose I could have and should have provided this info as well, but my goal was to produce a counterexample to refute the OP's hypothesis, thereby showing its invalidity Nevertheless, you should adjust your second sentence to be "In this case covariance and correlation matrices will be at most rank n−1, so there will be at least (p−n+1) zero eigenvalues." $\endgroup$ – Mark L. Stone Nov 21 '15 at 15:21
4
$\begingroup$

Consider $X$ to be an r.v. with mean 0 and variance of 1. Let $Y=2X$, and calculate the covariance matrix of $(X,Y)$. Since $2X=Y$, $E[Y^2]=4E[X^2]=\sigma_Y^2$, and $E[XY]=2E[X^2]$. Due to the zero mean configuration, the second moments are equal to the suitable covariances, for instance: $\mbox{Cov}(X,Y)=E[XY]-EXEY=E[XY]$.

So the covariance matrix will be: $$ \Lambda = \left(\array{1 & 2 \\ 2 & 4 }\right),$$ having a zero eigenvalue. The correlation matrix will be: $$ \Lambda = \left(\array{1 & 1 \\ 1 & 1 }\right),$$ having a zero eigenvalue as well. Due to the linear correspondence between $X$ and $Y$ it is easy to see why we get this correlation matrix - the diagonal will always be 1, and the off-diagonal is 1 because of the linear relationship.

$\endgroup$
  • $\begingroup$ Just for math-challenged readers like myself, let me point out that the 2 in $\Lambda$ is the $cov(X,Y)= \mathbb{E}(XY)-\mathbb{E}(X)\mathbb{E}(Y)= 2\mathbb{E}[X^2]=2\left(\sigma_X^2+[E(X)]^2\right)$ this last equality resulting from: $E(X^2)=\text{Var}(X)+[E(X)]^2$. $\endgroup$ – Antoni Parellada Nov 21 '15 at 19:45
  • $\begingroup$ +1 for your post. I wanted to make it easy to follow for everyone by including the $diag \Lambda^{-1/2}\,\Lambda\, diag \Lambda^{1/2}$ formula, but the comments format won't allow it. Do you think there is any valid point in including it in your post? $\endgroup$ – Antoni Parellada Nov 21 '15 at 20:44
  • $\begingroup$ @AntoniParellada , I'm not exactly sure what you mean - the covariance here is a direct calculation. But I will edit and make that clearer. Thanks. $\endgroup$ – yoki Nov 22 '15 at 14:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.