What does a non positive definite covariance matrix tell me about my data? I have a number of multivariate observations and would like to evaluate the probability density across all variables. It is assumed that the data is normally distributed. At low numbers of variables everything works as I would expect, but moving to greater numbers results in the covariance matrix becoming non positive definite.
I have reduced the problem in Matlab to:
load raw_data.mat; % matrix number-of-values x number of variables
Sigma = cov(data);
[R,err] = cholcov(Sigma, 0); % Test for pos-def done in mvnpdf.

If err>0 then Sigma is not positive definite.
Is there anything that I can do in order to evaluate my experimental data at higher dimensions? Does it tell me anything useful about my data?
I'm somewhat of a beginner in this area so apologies if I've missed out something obvious.
 A: The covariance matrix is not positive definite because it is singular.  That means that at least one of your variables can be expressed as a linear combination of the others.  You do not need all the variables as the value of at least one can be determined from a subset of the others.  I would suggest adding variables sequentially and checking the covariance matrix at each step.  If a new variable creates a singularity drop it and go on the the next one.  Eventually you should have a subset of variables with a postive definite covariance matrix.
A: One point that I don't think is addressed above is that it IS possible to calculate a non-positive definite covariance matrix from empirical data even if your variables are not perfectly linearly related. If you don't have sufficient data (particularly if you are trying to construct a high-dimensional covariance matrix from a bunch of pairwise comparisons) or if your data don't follow a multivariate normal distribution, then you can end up with paradoxical relationships among variables, such as cov(A,B)>0; cov(A,C)>0; cov(B,C)<0.
In such a case, one cannot fit a multivariate normal PDF, as there is no multivariate normal distribution that meets these criteria - cov(A,B)>0 and cov(A,C)>0 necessarily implies that cov(B,C)>0.
All this is to say, a non-positive definite matrix does not always mean that you are including collinear variables. It could also suggest that you are trying to model a relationship which is impossible given the parametric structure that you have chosen.
A: This post was a good read. I have seen this issue multiple times when using different functions in lavaan but was never aware what exactly it meant. This was also a very nice read on the subject if you want to peruse it. I feel it gives a great summary on what exactly a positive definite matrix is, why it ends up this way, and why it is problematic:
http://www3.nccu.edu.tw/~mnyu/2020%20Latent%20Variable%20Modeling/Not%20Positive%20Definite%20Matrices.pdf
