59
$\begingroup$

I guess the answer should be yes, but I still feel something is not right. There should be some general results in the literature, could anyone help me?

$\endgroup$
  • 2
    $\begingroup$ Every covariance matrix is Positive semi-definite. That means every covariance matrix must have non-negative eigen values. If none of the eigen value is zero then covariance matrix is additionally a Positive definite. $\endgroup$ – kaka May 29 '15 at 3:01
  • 3
  • 5
    $\begingroup$ @Jingjings: I can see in your profile that you have never upvoted or accepted any answers; this is quite remarkable given that you have a lot of good questions with many good answers. I guess you are not really aware of how it works. The idea is that you should upvote any answer that you find useful and accept any answer that you think solves your issue. It looks like you can upvote a lot of answers and also accept some of them. $\endgroup$ – amoeba Mar 3 '16 at 0:16
  • $\begingroup$ A=[4 1 0;1 0 -1 ;0 -1 8] is positive define $\endgroup$ – hemn1 morelany Jun 5 at 8:50
  • 1
    $\begingroup$ A valuable discussion on this topic can be found in our paper Stehlik, M.; Helpersdorfer Ch., Hermann,P. Supina,J. Grilo, L. M., Maidana; J.P., Fuders, F. Stehlikova, S. (2017) Financial and risk modelling with semicontinuous covariances, Information Sciences, 394-395, 246-272 $\endgroup$ – Milan Stehlik Aug 23 at 11:56
52
$\begingroup$

No.

Consider three variables, $X$, $Y$ and $Z = X+Y$. Their covariance matrix, $M$, is not positive definite, since there's a vector $z$ ($= (1, 1, -1)'$) for which $z'Mz$ is not positive.

Population covariance matrices are positive semi-definite.

(See property 2 here.)

The same should generally apply to covariance matrices of complete samples (no missing values), since they can also be seen as a form of discrete population covariance.

However due to inexactness of floating point numerical computations, even algebraically positive definite cases might occasionally be computed to not be even positive semi-definite; good choice of algorithms can help with this.

More generally, sample covariance matrices - depending on how they deal with missing values in some variables - may or may not be positive semi-definite, even in theory. If pairwise deletion is used, for example, then there's no guarantee of positive semi-definiteness. Further, accumulated numerical error can cause sample covariance matrices that should be notionally positive semi-definite to fail to be.

Like so:

 x <- rnorm(30)
 y <- rnorm(30) - x/10 # it doesn't matter for this if x and y are correlated or not
 z <- x+y
 M <- cov(data.frame(x=x,y=y,z=z))
 z <- rbind(1,1,-1)
 t(z)%*%M%*%z
              [,1]
[1,] -1.110223e-16

This happened on the first example I tried (I probably should supply a seed but it's not so rare that you should have to try a lot of examples before you get one).

The result came out negative, even though it should be algebraically zero. A different set of numbers might yield a positive number or an "exact" zero.

--

Example of moderate missingness leading to loss of positive semidefiniteness via pairwise deletion:

z <- x + y + rnorm(30)/50  # same x and y as before.
xyz1 <- data.frame(x=x,y=y,z=z) # high correlation but definitely of full rank 

xyz1$x[sample(1:30,5)] <- NA   # make 5 x's missing  

xyz1$y[sample(1:30,5)] <- NA   # make 5 y's missing  

xyz1$z[sample(1:30,5)] <- NA   # make 5 z's missing  

cov(xyz1,use="pairwise")     # the individual pairwise covars are fine ...

           x          y        z
x  1.2107760 -0.2552947 1.255868
y -0.2552947  1.2728156 1.037446
z  1.2558683  1.0374456 2.367978

 chol(cov(xyz1,use="pairwise"))  # ... but leave the matrix not positive semi-definite

Error in chol.default(cov(xyz1, use = "pairwise")) : 
  the leading minor of order 3 is not positive definite

 chol(cov(xyz1,use="complete")) # but deleting even more rows leaves it PSD

          x          y          z
x 0.8760209 -0.2253484 0.64303448
y 0.0000000  1.1088741 1.11270078
z 0.0000000  0.0000000 0.01345364
| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ +1: But as a comment mostly for you wording: As you present it, it looks like that PSD-ness is not guaranteed in the general case. As shown in sjm.majewski's answer you need a "pathological" case (non-full rank) and you end up with that issue. (I fully agree with the numerics comment) Can you elaborate a bit more missing values problem where you can't even guarantee PSD even if you account of numerical errors? (I assume you aren't concerned with sparsity of measurements etc. when saying that) $\endgroup$ – usεr11852 Apr 22 '13 at 14:08
  • 3
    $\begingroup$ Of course it only occurs when it's not of full rank (or very close to it). Look at the definition of PSD (and @sjm.majewski's mention of the relation to variance), and this much is clear. But to define it as pathological seems odd, since these non-full rank situations happen all the time in practice. This is not simple pedantry - it affects real data sets every day, and as a result generates regular questions here. I'll talk about missingness and pairwise deletion above, because there's not room for it here. $\endgroup$ – Glen_b Apr 22 '13 at 15:15
  • 3
    $\begingroup$ I think it would be great to add to this answer an explicit remark that in the $n<p$ situation the sample covariance matrix will be guaranteed to fail to be positive-definite (it will be low rank, i.e. will have some zero eigenvalues). I was searching if we have a thread that this Q stats.stackexchange.com/questions/198488 can be closed as a duplicate of, and I think this would be a good candidate but you don't seem to be mentioning the $n<p$ case. $\endgroup$ – amoeba Mar 3 '16 at 0:12
34
$\begingroup$

Well, to understand why the covariance matrix of a population is always positive semi-definite, notice that: $$ \sum_{i,j =1}^{n} y_i \cdot y_j \cdot Cov(X_i, X_j) = Var(\sum_{i=1}^n y_iX_i) \geq 0 $$ where $y_i$ are some real numbers, and $X_i$ are some real valued random variables.

This also explains why in the example given by Glen_b the covariance matrix was not positive definite . We had $y_1 =1 , y_2 = 1, y_3 = -1$, and $X_1 = X, X_2 = Y, X_3 = Z = X+Y$, so $\sum_{i=1}^{3} y_iX_i = 0$, and the variance of a random variable which is constant is $0$.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ This ought to be the accepted answer. The question just asks about "covariance matrices" which generally refer to the population covariance matrix of random variables, not the sample. $\endgroup$ – user3303 Oct 6 '16 at 19:41
  • $\begingroup$ May I ask what is the formula you used in your answer? $\endgroup$ – Aqqqq Jun 5 '18 at 6:07
  • $\begingroup$ If you mean the formula with variance and covariances, then you can derive it from the formula for the square of the sum (that is square of the sum is equal to the sum of products for all pairs). $\endgroup$ – sjm.majewski Jun 15 '18 at 20:55
  • $\begingroup$ where was this formula derived $\sum_{i,j =1}^{n} y_i \cdot y_j \cdot Cov(X_i, X_j) = Var(\sum_{i=1}^n y_iX_i) \geq 0$? $\endgroup$ – GENIVI-LEARNER Mar 28 at 17:35
1
$\begingroup$

As the other answers already make clear, a covariance matrix is not necessarily positive definite, but only positive semi-definite.

However, a covariance matrix is generally positive definite unless the space spanned by the variables is actually a linear subspace of lower dimension. This is exactly why in the example with X, Y and Z=X+Y the result is only positive semi-definite, but not positive definite. Although the variables span a three-dimensional space, they actually describe only a two-dimensional linear subspace (because they are not linearly independent).

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

As the other answer note, the covariance matrix is positive semi-definite (which I prefer to call non-negative definite), but not necessarily positive definite. We can show that the covariance matrix is positive semi-definite from first principles using its definition. To do this, suppose we consider a random vector $\mathbf{X}$ with mean vector $\boldsymbol{\mu}$ and covariance matrix $\mathbf{\Sigma}_\mathbf{X}$. For any conformable vector $\mathbf{z}$ we can define the corresponding vector:

$$\mathbf{Y} = (\mathbf{X} - \boldsymbol{\mu}_\mathbf{X})^\text{T} \mathbf{z}.$$

Since $||\mathbf{Y}|| \geqslant 0$ we then have:

$$\begin{aligned} \mathbf{z}^\text{T} \mathbf{\Sigma}_\mathbf{X} \mathbf{z} &= \mathbf{z}^\text{T} \mathbb{E}((\mathbf{X} - \boldsymbol{\mu}_\mathbf{X}) (\mathbf{X} - \boldsymbol{\mu}_\mathbf{X})^\text{T}) \mathbf{z} \\[6pt] &= \mathbb{E}(\mathbf{z}^\text{T} (\mathbf{X} - \boldsymbol{\mu}_\mathbf{X}) (\mathbf{X} - \boldsymbol{\mu}_\mathbf{X})^\text{T} \mathbf{z}) \\[6pt] &= \mathbb{E}(\mathbf{Y}^\text{T} \mathbf{Y}) \\[6pt] &= \mathbb{E}(||\mathbf{Y}||^2) \geqslant 0. \\[6pt] \end{aligned}$$

This establishes that the covariance matrix $\mathbf{\Sigma}_\mathbf{X}$ is positive semi-definite. Moreover, we can see that $\mathbf{z}^\text{T} \mathbf{\Sigma}_\mathbf{X} \mathbf{z} = 0$ if and only if $\mathbf{Y}=(\mathbf{X} - \boldsymbol{\mu}_\mathbf{X})^\text{T} \mathbf{z}=\mathbf{0}$ almost surely.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

$$\begin{array}{l}theory:\left\{ {{{\bf{\Sigma }}_{\bf{X}}}{\rm{ is positive semi - definite}}} \right.\\proof::\\set:\left\{ {{\bf{a}} = {\rm{vector }}\left( {p \times 1} \right){\rm{ }}\left( {{\mathop{\rm const}\nolimits} } \right) \ne \vec 0} \right.\\{{\bf{a}}^T}\Sigma {\bf{a}} = {\left[ {\begin{array}{*{20}{c}}{{a_1}}\\{{a_2}}\\ \vdots \\{{a_p}}\end{array}} \right]^T}\left[ {\begin{array}{*{20}{c}}{{\sigma _{11}}}&{{\sigma _{12}}}& \cdots &{{\sigma _{1p}}}\\{{\sigma _{21}}}&{{\sigma _{22}}}& \cdots &{{\sigma _{2p}}}\\ \vdots & \vdots & \ddots & \vdots \\{{\sigma _{p1}}}&{{\sigma _{p2}}}& \cdots &{{\sigma _{pp}}}\end{array}} \right]\left[ {\begin{array}{*{20}{c}}{{a_1}}\\{{a_2}}\\ \vdots \\{{a_p}}\end{array}} \right]\\ = \left[ {\begin{array}{*{20}{c}}{{a_1}{\sigma _{11}} + {a_2}{\sigma _{21}} + \cdots + {a_p}{\sigma _{p1}}}& \cdots & \cdots &{{a_1}{\sigma _{1p}} + {a_2}{\sigma _{2p}} + \cdots + {a_p}{\sigma _{pp}}}\end{array}} \right]\left[ {\begin{array}{*{20}{c}}{{a_1}}\\{{a_2}}\\ \vdots \\{{a_p}}\end{array}} \right]\\ = \left[ {\begin{array}{*{20}{c}}{\sum\limits_{i = 1}^p {{a_i}{\sigma _{i1}}} }& \cdots & \cdots &{\sum\limits_{i = 1}^p {{a_i}{\sigma _{ip}}} }\end{array}} \right]\left[ {\begin{array}{*{20}{c}}{{a_1}}\\{{a_2}}\\ \vdots \\{{a_p}}\end{array}} \right]\\ = \left[ {\begin{array}{*{20}{c}}{\sum\limits_{i = 1}^p {{a_i}{\mathop{\rm Cov}\nolimits} \left( {{X_i},{X_1}} \right)} }& \cdots & \cdots &{\sum\limits_{i = 1}^p {{a_i}{\mathop{\rm Cov}\nolimits} \left( {{X_i},{X_p}} \right)} }\end{array}} \right]\left[ {\begin{array}{*{20}{c}}{{a_1}}\\{{a_2}}\\ \vdots \\{{a_p}}\end{array}} \right]\\ = \sum\limits_{j = 1}^p {{a_j}\sum\limits_{i = 1}^p {{a_i}{\mathop{\rm Cov}\nolimits} \left( {{X_i},{X_j}} \right)} } = \sum\limits_{j = 1}^p {\sum\limits_{i = 1}^p {{a_i}{a_j}{\mathop{\rm Cov}\nolimits} \left( {{X_i},{X_j}} \right)} } \\\left[ {rule:\left\{ {{\mathop{\rm var}} \left( {\sum\limits_{i = 1}^p {{a_i}{X_i}} } \right) = \sum\limits_{j = 1}^p {\sum\limits_{i = 1}^p {{a_i}{a_j}{\mathop{\rm cov}} \left( {{X_i},{X_j}} \right)} } } \right.{\rm{ }}\left[ {see{\rm{ }}below} \right]} \right.\\ = {\mathop{\rm Var}\nolimits} \left( {\sum\limits_{i = 1}^p {{a_i}{X_i}} } \right) = {\mathop{\rm Var}\nolimits} \left( {{{\bf{a}}^T}{\bf{X}}} \right) \ge 0\end{array} $$


Following is followed by the answer from sjm.majewski

$$\begin{array}{l}rule:\left\{ {{\mathop{\rm var}} \left( {\sum\limits_{i = 1}^p {{a_i}{X_i}} } \right) = \sum\limits_{j = 1}^p {\sum\limits_{i = 1}^p {{a_i}{a_j}{\mathop{\rm cov}} \left( {{X_i},{X_j}} \right)} } } \right.\\proof,eg:\\\left\{ \begin{array}{l}{\mathop{\rm var}} \left( {{a_1}{X_1} + {a_2}{X_2} + {a_3}{X_3}} \right) = {\mathop{\rm var}} \left( {\sum\limits_{i = 1}^3 {{a_i}{X_i}} } \right)\\ = {\mathop{\rm var}} \left( {{a_1}{X_1} + {a_2}{X_2}} \right) + 2{\mathop{\rm cov}} \left( {{a_1}{X_1} + {a_2}{X_2},{a_3}{X_3}} \right) + {\mathop{\rm var}} \left( {{a_3}{X_3}} \right)\\ = \left( {{\mathop{\rm var}} \left( {{a_1}{X_1}} \right) + 2{\mathop{\rm cov}} \left( {{a_1}{X_1},{a_2}{X_2}} \right) + {\mathop{\rm var}} \left( {{a_2}{X_2}} \right)} \right) + 2\left( {{\mathop{\rm cov}} \left( {{a_1}{X_1},{a_3}{X_3}} \right) + {\mathop{\rm cov}} \left( {{a_2}{X_2},{a_3}{X_3}} \right)} \right) + {\mathop{\rm var}} \left( {{a_3}{X_3}} \right)\\ = \left( {{a_1}^2{\mathop{\rm var}} \left( {{X_1}} \right) + 2{a_1}{a_2}{\mathop{\rm cov}} \left( {{X_1},{X_2}} \right) + {a_2}^2{\mathop{\rm var}} \left( {{X_2}} \right)} \right) + 2\left( {{a_1}{a_3}{\mathop{\rm cov}} \left( {{X_1},{X_3}} \right) + {a_2}{a_3}{\mathop{\rm cov}} \left( {{X_2},{X_3}} \right)} \right) + {a_3}^2{\mathop{\rm var}} \left( {{X_3}} \right)\\ = {a_1}^2{\mathop{\rm var}} \left( {{X_1}} \right) + 2{a_1}{a_2}{\mathop{\rm cov}} \left( {{X_1},{X_2}} \right) + {a_2}^2{\mathop{\rm var}} \left( {{X_2}} \right) + 2{a_1}{a_3}{\mathop{\rm cov}} \left( {{X_1},{X_3}} \right) + 2{a_2}{a_3}{\mathop{\rm cov}} \left( {{X_2},{X_3}} \right) + {a_3}^2{\mathop{\rm var}} \left( {{X_3}} \right)\\ = \sum\limits_{j = 1}^3 {\sum\limits_{i = 1}^3 {{a_i}{a_j}{\mathop{\rm Cov}\nolimits} \left( {{X_i},{X_j}} \right)} } \end{array} \right.\end{array}% $$

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.