50
$\begingroup$

I guess the answer should be yes, but I still feel something is not right. There should be some general results in the literature, could anyone help me?

$\endgroup$
47
$\begingroup$

No.

Consider three variables, $X$, $Y$ and $Z = X+Y$. Their covariance matrix, $M$, is not positive definite, since there's a vector $z$ ($= (1, 1, -1)'$) for which $z'Mz$ is not positive.

Population covariance matrices are positive semi-definite.

(See property 2 here.)

The same should generally apply to covariance matrices of complete samples (no missing values), since they can also be seen as a form of discrete population covariance.

However due to inexactness of floating point numerical computations, even algebraically positive definite cases might occasionally be computed to not be even positive semi-definite; good choice of algorithms can help with this.

More generally, sample covariance matrices - depending on how they deal with missing values in some variables - may or may not be positive semi-definite, even in theory. If pairwise deletion is used, for example, then there's no guarantee of positive semi-definiteness. Further, accumulated numerical error can cause sample covariance matrices that should be notionally positive semi-definite to fail to be.

Like so:

 x <- rnorm(30)
 y <- rnorm(30) - x/10 # it doesn't matter for this if x and y are correlated or not
 z <- x+y
 M <- cov(data.frame(x=x,y=y,z=z))
 z <- rbind(1,1,-1)
 t(z)%*%M%*%z
              [,1]
[1,] -1.110223e-16

This happened on the first example I tried (I probably should supply a seed but it's not so rare that you should have to try a lot of examples before you get one).

The result came out negative, even though it should be algebraically zero. A different set of numbers might yield a positive number or an "exact" zero.

--

Example of moderate missingness leading to loss of positive semidefiniteness via pairwise deletion:

z <- x + y + rnorm(30)/50  # same x and y as before.
xyz1 <- data.frame(x=x,y=y,z=z) # high correlation but definitely of full rank 

xyz1$x[sample(1:30,5)] <- NA   # make 5 x's missing  

xyz1$y[sample(1:30,5)] <- NA   # make 5 y's missing  

xyz1$z[sample(1:30,5)] <- NA   # make 5 z's missing  

cov(xyz1,use="pairwise")     # the individual pairwise covars are fine ...

           x          y        z
x  1.2107760 -0.2552947 1.255868
y -0.2552947  1.2728156 1.037446
z  1.2558683  1.0374456 2.367978

 chol(cov(xyz1,use="pairwise"))  # ... but leave the matrix not positive semi-definite

Error in chol.default(cov(xyz1, use = "pairwise")) : 
  the leading minor of order 3 is not positive definite

 chol(cov(xyz1,use="complete")) # but deleting even more rows leaves it PSD

          x          y          z
x 0.8760209 -0.2253484 0.64303448
y 0.0000000  1.1088741 1.11270078
z 0.0000000  0.0000000 0.01345364
$\endgroup$
  • 1
    $\begingroup$ +1: But as a comment mostly for you wording: As you present it, it looks like that PSD-ness is not guaranteed in the general case. As shown in sjm.majewski's answer you need a "pathological" case (non-full rank) and you end up with that issue. (I fully agree with the numerics comment) Can you elaborate a bit more missing values problem where you can't even guarantee PSD even if you account of numerical errors? (I assume you aren't concerned with sparsity of measurements etc. when saying that) $\endgroup$ – usεr11852 says Reinstate Monic Apr 22 '13 at 14:08
  • 2
    $\begingroup$ Of course it only occurs when it's not of full rank (or very close to it). Look at the definition of PSD (and @sjm.majewski's mention of the relation to variance), and this much is clear. But to define it as pathological seems odd, since these non-full rank situations happen all the time in practice. This is not simple pedantry - it affects real data sets every day, and as a result generates regular questions here. I'll talk about missingness and pairwise deletion above, because there's not room for it here. $\endgroup$ – Glen_b -Reinstate Monica Apr 22 '13 at 15:15
  • 2
    $\begingroup$ I think it would be great to add to this answer an explicit remark that in the $n<p$ situation the sample covariance matrix will be guaranteed to fail to be positive-definite (it will be low rank, i.e. will have some zero eigenvalues). I was searching if we have a thread that this Q stats.stackexchange.com/questions/198488 can be closed as a duplicate of, and I think this would be a good candidate but you don't seem to be mentioning the $n<p$ case. $\endgroup$ – amoeba says Reinstate Monica Mar 3 '16 at 0:12
27
$\begingroup$

Well, to understand why the covariance matrix of a population is always positive semi-definite, notice that: $$ \sum_{i,j =1}^{n} y_i \cdot y_j \cdot Cov(X_i, X_j) = Var(\sum_{i=1}^n y_iX_i) \geq 0 $$ where $y_i$ are some real numbers, and $X_i$ are some real valued random variables.

This also explains why in the example given by Glen_b the covariance matrix was not positive definite . We had $y_1 =1 , y_2 = 1, y_3 = -1$, and $X_1 = X, X_2 = Y, X_3 = Z = X+Y$, so $\sum_{i=1}^{3} y_iX_i = 0$, and the variance of a random variable which is constant is $0$.

$\endgroup$
  • 1
    $\begingroup$ This ought to be the accepted answer. The question just asks about "covariance matrices" which generally refer to the population covariance matrix of random variables, not the sample. $\endgroup$ – user3303 Oct 6 '16 at 19:41
  • $\begingroup$ May I ask what is the formula you used in your answer? $\endgroup$ – Aqqqq Jun 5 '18 at 6:07
  • $\begingroup$ If you mean the formula with variance and covariances, then you can derive it from the formula for the square of the sum (that is square of the sum is equal to the sum of products for all pairs). $\endgroup$ – sjm.majewski Jun 15 '18 at 20:55
0
$\begingroup$

As the other answers already make clear, a covariance matrix is not necessarily positive definite, but only positive semi-definite.

However, a covariance matrix is generally positive definite unless the space spanned by the variables is actually a linear subspace of lower dimension. This is exactly why in the example with X, Y and Z=X+Y the result is only positive semi-definite, but not positive definite. Although the variables span a three-dimensional space, they actually describe only a two-dimensional linear subspace (because they are not linearly independent).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.