I guess the answer should be yes, but I still feel something is not right. There should be some general results in the literature, could anyone help me?

  • 2
    $\begingroup$ Every covariance matrix is Positive semi-definite. That means every covariance matrix must have non-negative eigen values. If none of the eigen value is zero then covariance matrix is additionally a Positive definite. $\endgroup$
    – kaka
    May 29, 2015 at 3:01
  • 3
  • 7
    $\begingroup$ @Jingjings: I can see in your profile that you have never upvoted or accepted any answers; this is quite remarkable given that you have a lot of good questions with many good answers. I guess you are not really aware of how it works. The idea is that you should upvote any answer that you find useful and accept any answer that you think solves your issue. It looks like you can upvote a lot of answers and also accept some of them. $\endgroup$
    – amoeba
    Mar 3, 2016 at 0:16
  • $\begingroup$ A=[4 1 0;1 0 -1 ;0 -1 8] is positive define $\endgroup$ Jun 5, 2020 at 8:50
  • 1
    $\begingroup$ A valuable discussion on this topic can be found in our paper Stehlik, M.; Helpersdorfer Ch., Hermann,P. Supina,J. Grilo, L. M., Maidana; J.P., Fuders, F. Stehlikova, S. (2017) Financial and risk modelling with semicontinuous covariances, Information Sciences, 394-395, 246-272 $\endgroup$ Aug 23, 2020 at 11:56

5 Answers 5



Consider three variables, $X$, $Y$ and $Z = X+Y$. Their covariance matrix, $M$, is not positive definite, since there's a vector $z$ ($= (1, 1, -1)'$) for which $z'Mz$ is not positive.

Population covariance matrices are positive semi-definite.

(See property 2 here.)

The same should generally apply to covariance matrices of complete samples (no missing values), since they can also be seen as a form of discrete population covariance.

However due to inexactness of floating point numerical computations, even algebraically positive definite cases might occasionally be computed to not be even positive semi-definite; good choice of algorithms can help with this.

More generally, sample covariance matrices - depending on how they deal with missing values in some variables - may or may not be positive semi-definite, even in theory. If pairwise deletion is used, for example, then there's no guarantee of positive semi-definiteness. Further, accumulated numerical error can cause sample covariance matrices that should be notionally positive semi-definite to fail to be.

Like so:

 x <- rnorm(30)
 y <- rnorm(30) - x/10 # it doesn't matter for this if x and y are correlated or not
 z <- x+y
 M <- cov(data.frame(x=x,y=y,z=z))
 z <- rbind(1,1,-1)
[1,] -1.110223e-16

This happened on the first example I tried (I probably should supply a seed but it's not so rare that you should have to try a lot of examples before you get one).

The result came out negative, even though it should be algebraically zero. A different set of numbers might yield a positive number or an "exact" zero.


Example of moderate missingness leading to loss of positive semidefiniteness via pairwise deletion:

z <- x + y + rnorm(30)/50  # same x and y as before.
xyz1 <- data.frame(x=x,y=y,z=z) # high correlation but definitely of full rank 

xyz1$x[sample(1:30,5)] <- NA   # make 5 x's missing  

xyz1$y[sample(1:30,5)] <- NA   # make 5 y's missing  

xyz1$z[sample(1:30,5)] <- NA   # make 5 z's missing  

cov(xyz1,use="pairwise")     # the individual pairwise covars are fine ...

           x          y        z
x  1.2107760 -0.2552947 1.255868
y -0.2552947  1.2728156 1.037446
z  1.2558683  1.0374456 2.367978

 chol(cov(xyz1,use="pairwise"))  # ... but leave the matrix not positive semi-definite

Error in chol.default(cov(xyz1, use = "pairwise")) : 
  the leading minor of order 3 is not positive definite

 chol(cov(xyz1,use="complete")) # but deleting even more rows leaves it PSD

          x          y          z
x 0.8760209 -0.2253484 0.64303448
y 0.0000000  1.1088741 1.11270078
z 0.0000000  0.0000000 0.01345364
  • 1
    $\begingroup$ +1: But as a comment mostly for you wording: As you present it, it looks like that PSD-ness is not guaranteed in the general case. As shown in sjm.majewski's answer you need a "pathological" case (non-full rank) and you end up with that issue. (I fully agree with the numerics comment) Can you elaborate a bit more missing values problem where you can't even guarantee PSD even if you account of numerical errors? (I assume you aren't concerned with sparsity of measurements etc. when saying that) $\endgroup$
    – usεr11852
    Apr 22, 2013 at 14:08
  • 3
    $\begingroup$ Of course it only occurs when it's not of full rank (or very close to it). Look at the definition of PSD (and @sjm.majewski's mention of the relation to variance), and this much is clear. But to define it as pathological seems odd, since these non-full rank situations happen all the time in practice. This is not simple pedantry - it affects real data sets every day, and as a result generates regular questions here. I'll talk about missingness and pairwise deletion above, because there's not room for it here. $\endgroup$
    – Glen_b
    Apr 22, 2013 at 15:15
  • 3
    $\begingroup$ I think it would be great to add to this answer an explicit remark that in the $n<p$ situation the sample covariance matrix will be guaranteed to fail to be positive-definite (it will be low rank, i.e. will have some zero eigenvalues). I was searching if we have a thread that this Q stats.stackexchange.com/questions/198488 can be closed as a duplicate of, and I think this would be a good candidate but you don't seem to be mentioning the $n<p$ case. $\endgroup$
    – amoeba
    Mar 3, 2016 at 0:12

Well, to understand why the covariance matrix of a population is always positive semi-definite, notice that: $$ \sum_{i,j =1}^{n} y_i \cdot y_j \cdot Cov(X_i, X_j) = Var(\sum_{i=1}^n y_iX_i) \geq 0 $$ where $y_i$ are some real numbers, and $X_i$ are some real valued random variables.

This also explains why in the example given by Glen_b the covariance matrix was not positive definite . We had $y_1 =1 , y_2 = 1, y_3 = -1$, and $X_1 = X, X_2 = Y, X_3 = Z = X+Y$, so $\sum_{i=1}^{3} y_iX_i = 0$, and the variance of a random variable which is constant is $0$.

  • $\begingroup$ May I ask what is the formula you used in your answer? $\endgroup$
    – Aqqqq
    Jun 5, 2018 at 6:07
  • $\begingroup$ If you mean the formula with variance and covariances, then you can derive it from the formula for the square of the sum (that is square of the sum is equal to the sum of products for all pairs). $\endgroup$ Jun 15, 2018 at 20:55
  • 1
    $\begingroup$ where was this formula derived $\sum_{i,j =1}^{n} y_i \cdot y_j \cdot Cov(X_i, X_j) = Var(\sum_{i=1}^n y_iX_i) \geq 0$? $\endgroup$ Mar 28, 2020 at 17:35

As the other answer note, the covariance matrix is positive semi-definite (which I prefer to call non-negative definite), but not necessarily positive definite. We can show that the covariance matrix is positive semi-definite from first principles using its definition. To do this, suppose we consider a random vector $\mathbf{X}$ with mean vector $\boldsymbol{\mu}$ and covariance matrix $\mathbf{\Sigma}_\mathbf{X}$. For any conformable vector $\mathbf{z}$ we can define the corresponding vector:

$$\mathbf{Y} = (\mathbf{X} - \boldsymbol{\mu}_\mathbf{X})^\text{T} \mathbf{z}.$$

Since $||\mathbf{Y}|| \geqslant 0$ we then have:

$$\begin{aligned} \mathbf{z}^\text{T} \mathbf{\Sigma}_\mathbf{X} \mathbf{z} &= \mathbf{z}^\text{T} \mathbb{E}((\mathbf{X} - \boldsymbol{\mu}_\mathbf{X}) (\mathbf{X} - \boldsymbol{\mu}_\mathbf{X})^\text{T}) \mathbf{z} \\[6pt] &= \mathbb{E}(\mathbf{z}^\text{T} (\mathbf{X} - \boldsymbol{\mu}_\mathbf{X}) (\mathbf{X} - \boldsymbol{\mu}_\mathbf{X})^\text{T} \mathbf{z}) \\[6pt] &= \mathbb{E}(\mathbf{Y}^\text{T} \mathbf{Y}) \\[6pt] &= \mathbb{E}(||\mathbf{Y}||^2) \geqslant 0. \\[6pt] \end{aligned}$$

This establishes that the covariance matrix $\mathbf{\Sigma}_\mathbf{X}$ is positive semi-definite. Moreover, we can see that $\mathbf{z}^\text{T} \mathbf{\Sigma}_\mathbf{X} \mathbf{z} = 0$ if and only if $\mathbf{Y}=(\mathbf{X} - \boldsymbol{\mu}_\mathbf{X})^\text{T} \mathbf{z}=\mathbf{0}$ almost surely.


As the other answers already make clear, a covariance matrix is not necessarily positive definite, but only positive semi-definite.

However, a covariance matrix is generally positive definite unless the space spanned by the variables is actually a linear subspace of lower dimension. This is exactly why in the example with X, Y and Z=X+Y the result is only positive semi-definite, but not positive definite. Although the variables span a three-dimensional space, they actually describe only a two-dimensional linear subspace (because they are not linearly independent).


$$\begin{array}{l}theory:\left\{ {{{\bf{\Sigma }}_{\bf{X}}}{\rm{ is positive semi - definite}}} \right.\\proof::\\set:\left\{ {{\bf{a}} = {\rm{vector }}\left( {p \times 1} \right){\rm{ }}\left( {{\mathop{\rm const}\nolimits} } \right) \ne \vec 0} \right.\\{{\bf{a}}^T}\Sigma {\bf{a}} = {\left[ {\begin{array}{*{20}{c}}{{a_1}}\\{{a_2}}\\ \vdots \\{{a_p}}\end{array}} \right]^T}\left[ {\begin{array}{*{20}{c}}{{\sigma _{11}}}&{{\sigma _{12}}}& \cdots &{{\sigma _{1p}}}\\{{\sigma _{21}}}&{{\sigma _{22}}}& \cdots &{{\sigma _{2p}}}\\ \vdots & \vdots & \ddots & \vdots \\{{\sigma _{p1}}}&{{\sigma _{p2}}}& \cdots &{{\sigma _{pp}}}\end{array}} \right]\left[ {\begin{array}{*{20}{c}}{{a_1}}\\{{a_2}}\\ \vdots \\{{a_p}}\end{array}} \right]\\ = \left[ {\begin{array}{*{20}{c}}{{a_1}{\sigma _{11}} + {a_2}{\sigma _{21}} + \cdots + {a_p}{\sigma _{p1}}}& \cdots & \cdots &{{a_1}{\sigma _{1p}} + {a_2}{\sigma _{2p}} + \cdots + {a_p}{\sigma _{pp}}}\end{array}} \right]\left[ {\begin{array}{*{20}{c}}{{a_1}}\\{{a_2}}\\ \vdots \\{{a_p}}\end{array}} \right]\\ = \left[ {\begin{array}{*{20}{c}}{\sum\limits_{i = 1}^p {{a_i}{\sigma _{i1}}} }& \cdots & \cdots &{\sum\limits_{i = 1}^p {{a_i}{\sigma _{ip}}} }\end{array}} \right]\left[ {\begin{array}{*{20}{c}}{{a_1}}\\{{a_2}}\\ \vdots \\{{a_p}}\end{array}} \right]\\ = \left[ {\begin{array}{*{20}{c}}{\sum\limits_{i = 1}^p {{a_i}{\mathop{\rm Cov}\nolimits} \left( {{X_i},{X_1}} \right)} }& \cdots & \cdots &{\sum\limits_{i = 1}^p {{a_i}{\mathop{\rm Cov}\nolimits} \left( {{X_i},{X_p}} \right)} }\end{array}} \right]\left[ {\begin{array}{*{20}{c}}{{a_1}}\\{{a_2}}\\ \vdots \\{{a_p}}\end{array}} \right]\\ = \sum\limits_{j = 1}^p {{a_j}\sum\limits_{i = 1}^p {{a_i}{\mathop{\rm Cov}\nolimits} \left( {{X_i},{X_j}} \right)} } = \sum\limits_{j = 1}^p {\sum\limits_{i = 1}^p {{a_i}{a_j}{\mathop{\rm Cov}\nolimits} \left( {{X_i},{X_j}} \right)} } \\\left[ {rule:\left\{ {{\mathop{\rm var}} \left( {\sum\limits_{i = 1}^p {{a_i}{X_i}} } \right) = \sum\limits_{j = 1}^p {\sum\limits_{i = 1}^p {{a_i}{a_j}{\mathop{\rm cov}} \left( {{X_i},{X_j}} \right)} } } \right.{\rm{ }}\left[ {see{\rm{ }}below} \right]} \right.\\ = {\mathop{\rm Var}\nolimits} \left( {\sum\limits_{i = 1}^p {{a_i}{X_i}} } \right) = {\mathop{\rm Var}\nolimits} \left( {{{\bf{a}}^T}{\bf{X}}} \right) \ge 0\end{array} $$

Following is followed by the answer from sjm.majewski

$$\begin{array}{l}rule:\left\{ {{\mathop{\rm var}} \left( {\sum\limits_{i = 1}^p {{a_i}{X_i}} } \right) = \sum\limits_{j = 1}^p {\sum\limits_{i = 1}^p {{a_i}{a_j}{\mathop{\rm cov}} \left( {{X_i},{X_j}} \right)} } } \right.\\proof,eg:\\\left\{ \begin{array}{l}{\mathop{\rm var}} \left( {{a_1}{X_1} + {a_2}{X_2} + {a_3}{X_3}} \right) = {\mathop{\rm var}} \left( {\sum\limits_{i = 1}^3 {{a_i}{X_i}} } \right)\\ = {\mathop{\rm var}} \left( {{a_1}{X_1} + {a_2}{X_2}} \right) + 2{\mathop{\rm cov}} \left( {{a_1}{X_1} + {a_2}{X_2},{a_3}{X_3}} \right) + {\mathop{\rm var}} \left( {{a_3}{X_3}} \right)\\ = \left( {{\mathop{\rm var}} \left( {{a_1}{X_1}} \right) + 2{\mathop{\rm cov}} \left( {{a_1}{X_1},{a_2}{X_2}} \right) + {\mathop{\rm var}} \left( {{a_2}{X_2}} \right)} \right) + 2\left( {{\mathop{\rm cov}} \left( {{a_1}{X_1},{a_3}{X_3}} \right) + {\mathop{\rm cov}} \left( {{a_2}{X_2},{a_3}{X_3}} \right)} \right) + {\mathop{\rm var}} \left( {{a_3}{X_3}} \right)\\ = \left( {{a_1}^2{\mathop{\rm var}} \left( {{X_1}} \right) + 2{a_1}{a_2}{\mathop{\rm cov}} \left( {{X_1},{X_2}} \right) + {a_2}^2{\mathop{\rm var}} \left( {{X_2}} \right)} \right) + 2\left( {{a_1}{a_3}{\mathop{\rm cov}} \left( {{X_1},{X_3}} \right) + {a_2}{a_3}{\mathop{\rm cov}} \left( {{X_2},{X_3}} \right)} \right) + {a_3}^2{\mathop{\rm var}} \left( {{X_3}} \right)\\ = {a_1}^2{\mathop{\rm var}} \left( {{X_1}} \right) + 2{a_1}{a_2}{\mathop{\rm cov}} \left( {{X_1},{X_2}} \right) + {a_2}^2{\mathop{\rm var}} \left( {{X_2}} \right) + 2{a_1}{a_3}{\mathop{\rm cov}} \left( {{X_1},{X_3}} \right) + 2{a_2}{a_3}{\mathop{\rm cov}} \left( {{X_2},{X_3}} \right) + {a_3}^2{\mathop{\rm var}} \left( {{X_3}} \right)\\ = \sum\limits_{j = 1}^3 {\sum\limits_{i = 1}^3 {{a_i}{a_j}{\mathop{\rm Cov}\nolimits} \left( {{X_i},{X_j}} \right)} } \end{array} \right.\end{array}% $$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.