# How to do factor analysis when the covariance matrix is not positive definite?

I have a data set that consists of 717 observations (rows) which are described by 33 variables (columns). The data are standardized by z-scoring all the variables. No two variables are linearly dependent ($r=1$). I've also removed all the variables with very low variance (less than $0.1$). The figure below shows the corresponding correlation matrix (in absolute values).

When I'm trying to run factor analysis using factoran in Matlab as follows:

[Loadings1,specVar1,T,stats] = factoran(Z2,1);


The data X must have a covariance matrix that is positive definite.


Could you please tell me where is the problem? Is it due to low mutual dependency among the used variables? In addition, what can I do about it?

My correlation matrix: • I think it is the contrary of low mutual dependency that is a problem here. You probably have some variables that are linearly dependent among themselves and this cause your covariance matrix to be semi-definite (ie. have some zero eigenvaues). – usεr11852 says Reinstate Monic Apr 7 '15 at 21:58
• To the people voting down the question to close: Why would a question on the positive-definiteness of sample covariance matrix be off-topic here? The user is concerned why a standard application of Factor Analysis does not work. Ask for more info if you want! – usεr11852 says Reinstate Monic Apr 8 '15 at 0:15
• Can you please compute and present the eigenvalues of the sample covariance matrix? (eg. eig(cov(Z2))). I strongly suspect that some of them are very small. – usεr11852 says Reinstate Monic Apr 8 '15 at 0:19
• I agree with @usεr11852: it seems that this question was wrongly closed as off-topic (I voted to close it myself). It did look like a programming question, but it is actually a perfectly on-topic and reasonable question. I edited it and voted to reopen. Pity that the OP seems to have disappeared though. – amoeba says Reinstate Monica Oct 2 '15 at 14:27
• I would argue there may be statistical content to this question that the Matlab community won't be able to help with. How do you compute your Z2 matrix? If you have missing values in your data, then pairwise deletion may drive the matrix to become noninvertible when the different correlations in that matrix are computed using different subsamples of the data. – StasK Oct 2 '15 at 14:31

Let's define the correlation matrix by $$C$$. Since it is positive semi-definite, but not positive definite, its spectral decomposition looks something like $$C = Q D Q^{-1},$$ where the columns of $$Q$$ consist of orthonormal eigenvectors of $$C$$ and $$D = \begin{pmatrix}\lambda_1 & 0 & \cdots & \cdots &\cdots & \cdots& 0\\ 0 & \lambda_2 & \ddots & && &\vdots \\ \vdots & \ddots &\ddots & \ddots && &\vdots \\ \vdots & &\ddots &\lambda_n &\ddots &&\vdots \\ \vdots & & & \ddots &0 & \ddots& \vdots \\ \vdots & & & &\ddots & \ddots& 0\\ 0 & \cdots &\cdots & \cdots &\cdots & 0& 0\end{pmatrix}$$ is a diagonal matrix containing the eigenvalues corresponding to the eigenvectors in $$Q$$. Some of those are $$0$$. Moreover, $$n$$ is the rank of $$C$$.

A simple way to restore positive definiteness is setting the $$0$$-eigenvalues to some value that is numerically non-zero, e.g. $$\lambda_{n+1}, \lambda_{n+2},... = 10^{-15}.$$ Hence, set $$\tilde{C} = Q \tilde{D} Q^{-1},$$ where $$\tilde{D} = \begin{pmatrix}\lambda_1 & 0 & \cdots & \cdots &\cdots & \cdots& 0\\ 0 & \lambda_2 & \ddots & && &\vdots \\ \vdots & \ddots &\ddots & \ddots && &\vdots \\ \vdots & &\ddots &\lambda_n &\ddots &&\vdots \\ \vdots & & & \ddots &10^{-15} & \ddots& \vdots \\ \vdots & & & &\ddots & \ddots& 0\\ 0 & \cdots &\cdots & \cdots &\cdots & 0& 10^{-15}\end{pmatrix}$$ Then, perform the factor analysis for \$\tilde{C}.

In Matlab, one can obtain $$Q,D$$ using the command:

[Q,D] = eig(C)


Constructing $$\tilde{C}$$ is then just simple Matrix manipulations.

Remark: It would be hard to tell how this influences the factor analysis though; hence, one should probably be careful with this method. Moreover, even though this is a $$C$$ is a correlation matrix, $$\tilde{C}$$ may well be not. Hence, another normalisation of the entries might be necessary.

It is possible that you are facing numeric issues with your matrix. It is possibly actually positive definite, but the numerical computation says otherwise.

A very common solution, in that case, is to add a very low value (1.E-10 for instance) to all diagonal elements. If this does not solve the problem, try to progressively increase this value.

• The diagonal entries seem pretty dominant already. Do you think this would help? – Jonas Dec 8 '18 at 22:44
• I am worried about the 8th and 10th entries, if some are linearly dependent, it must be these two. I don't know enough about the actual eigen decomposition computation, but I think that this solution could work: adding 1e-10 to the diagonal does not do much about the linear dependency, but it might just add whatever is numerically necessary so that the computation of eigen values is fixed (i.e. no 0 eigen value). Yet, if my solution does not work, I find yours quite elegant. – Romain Reboulleau Dec 11 '18 at 12:33

FA works best when your data is Gaussian, therefore you may want to try some pre-processing approaches to have a more Gaussian-like data.

• I'm bewildered by this answer. What is the sense in doing a FA of uncorrelated data? – ttnphns May 24 '18 at 2:12
• @ttnphns I think you're right! It does not make any sense to apply FA on decorrelated data! My suggestion comes from a specific kind of FA, where the PCA applied in a pre-processing of creating vector representations from data, in which you will apply FA on. The original data is usually projected by PCA before transformed into the vector representation. And the FA is applied on the vector representation, and not the PCA projected data. My bad! I'll update my answer. Although in some cases a probabilistic PCA might work as good as FA if the FA model failed to converge. Don't you agree? – PickleRick May 24 '18 at 4:35