Why is a sample covariance not even semi-positive definite with missing data?

Question

I am trying to estimate a sample covariance when I have less observations $n$ than variables $p$ ($n<p$). This will serve later on as basis for a shrinkage estimator.

We know (see this post) that the sample covariance with $n<p$ will not be positive-definite, but at least it should be semi-positive definite. Now I realized that if I have even a single missing value, and use the pairwise.complete.obs strategy as in R stats::cov (which computes individual entries with available data) the resulting matrix is not even semi-positive definite (i.e. there is a negative eigenvalue). See code below.

What is the intuition/mathematical phenomenon for the fact that my matrix is not more at least semi-positive definite?

Thanks!

X <- matrix(rnorm(500), nrow=5, ncol=10)
X_NA <- X
X_NA[2,1] <- NA

min(eigen(cov(X))$values)
#> [1] -3.440114e-16
min(eigen(cov(X_NA, use = "pairwise.complete.obs"))$values)
#> [1] -0.157207

^{Created on 2019-11-13 by the reprex package (v0.3.0)}

Answer 1

Why do you expect the sample covariance matrix to be semiposdef (positive semi-definite) when it is calculated in a nonstandard way, by omitting missing data via the pairwise.complete.obs strategy? The proof of semiposdef-ness assumes complete data, and is not valid for your case! If you need an estimated covariance matrix which is semiposdef, use imputation (or multiple imputation) on your dataset first, and then the usual sample covariance matrix.