"matrix is not positive definite" - even when highly correlated variables are removed

Question

I am running a factor analysis in SPSS and get a "matrix is not positive definite" error from my correlation matrix. I've tried removing correlated variables, but I have to remove all variables down to those with correlations of r = 0.8 before the "positive definite" issue is resolved. This seems weird, as I have read that this problem normally arises where two or more vbls are nearly perfectly collinear (to me, r = 0.8 is a high correlation, but not near-perfect.) Can anyone suggest a way of dealing with this issue that doesn't just involve arbitrary removal of variables until the problem goes away? Or perhaps explain why I am getting the issue even when I seem to be removing all the really highly correlated variables?

[edit by Gottfried Helms:] By comment to an answer the OP says, there are 14 variables on 10 cases per correlation.

Answer 1

The best tool to resolve (multi-) collinearity is in my view the Cholesky-decomposition of the correlation/covariance matrix. The following example discusses even the case of collinearity, where none of the bivariate correlations are "extreme", because we have rank-reduction only over sets of more variables than only two.

If the correlation-matrix, say R, is positive definite, then all entries on the diagonal of the cholesky-factor, say L, are non-zero (aka machine-epsilon). Btw, to use this tool for the collinearity-detection it must be implemented as to allow zero-eigenvalues, don't know, whether, for instance, you can use SPSS for this.
The number of non-zero entries in the diagonal indicate the actual rank of the correlation-matrix. And because of the triangular structure of the L-matrix the variables above the first occuring diagonal zero form a partial set of variables which is of reduced-rank. However, there may be some variables in that block, which do not belong to that set. So to find the crucial subset which contains only the multicollinearity you do several recomputations of the cholesky-decomposition, where you reorder the variables such that you find the smallest possible subset, which shows rank-reduction - so this is an iterative procedure. (If needed, I'll show an example where I use my MatMate-program for the script, later).

---

Here is an example using random-data on 5 variables, say $x_1$ to $x_5$ which I configured, such that the correlation matrix is positive semidefinite (up to machine precision) because I made $x_5 = 2 \cdot x_2 + \sqrt 2 \cdot x_4 $ (and after that normed to unit-variance) and thus that subset of three variables make a collinear subspace (more exactly: we should call it "co-planar" since they are linearly dependent only in a plane). Here is the correlation-matrix R

;MatMate-Listing vom:06.03.2013 17:43:23
;============================================

R =        x1        x2        x3        x4        x5
------------------------------------------------------
x1        1.0000   -0.7506    0.2298   -0.8666    0.0952
x2       -0.7506    1.0000   -0.2696    0.4569    0.5355
x3        0.2298   -0.2696    1.0000    0.1890   -0.4407
x4       -0.8666    0.4569    0.1890    1.0000   -0.5066
x5        0.0952    0.5355   -0.4407   -0.5066    1.0000
------------------------------------------------------

and here the cholesky-factor / loadingsmatrix:

[20]     L = cholesky(R)

L=        f1        f2        f3        f4        f5
------------------------------------------------------
x1        1.0000     .         .         .         .    
x2       -0.7506    0.6607     .         .         .    
x3        0.2298   -0.1469    0.9621     .         .    
x4       -0.8666   -0.2930    0.3587    0.1856     .    
x5        0.0952    0.9186   -0.3406   -0.1762     .    
------------------------------------------------------

As we see that only 4 of 5 diagonal elements are non-zero (above machine-epsilon) we know, that the correlation matrix has rank 4 instead of 5 and we have collinearity. But we do not yet know, whether 4 variables are linearly dependent or whether we have possibly a rank reduced subspace of even smaller dimension. So we try iteratively the rotation to triangularity, where the order of the variables $x_1$ to $x_5$ is systematically altered to identify any possible smallest subset.

For instance, we make the last item "the first"

[22] l1=rot(L,"drei",5´1´2´3´4)

L1=       f1        f2        f3        f4        f5
------------------------------------------------------
x1        0.0952    0.9955     .         .         .    
x2        0.5355   -0.8053   0.2545      .         .    
x3       -0.4407    0.2730   0.7320     0.4421     .    
x4       -0.5066   -0.8221   0.2598      .         .    
x5        1.0000     .         .         .         .    
------------------------------------------------------

and we see that rank-reduction is already occuring if we ignore variable 3 - because the variables $x_1,x_2,x_4,x_5$ define already a 3-dimensional subspace (instead of a 4-dimensional one).

Now we proceed altering the order for the cholesky-decomposition (actually I do this by a column rotation with a "triangularity-criterion"):

[24] L1 = rot(L,"drei",5´4´1´2´3)

L1=       f1        f2        f3        f4        f5
------------------------------------------------------
x1        0.0952   -0.9492    0.3000     .         .    
x2        0.5355    0.8445     .         .         .    
x3       -0.4407   -0.0397    0.7803     .        0.4421
x4       -0.5066    0.8622     .         .         .    
x5        1.0000     .         .         .         .    
------------------------------------------------------

Now we're nearly done: the subset of $x_2,x_4,x_5$ forms a reduced subspace and to see more, we put them at "the top" of the cholesky-process:

[26] L1 = rot(L,"drei",5´4´2´1´3)

L1=       f1        f2        f3        f4        f5
------------------------------------------------------
x1        0.0952   -0.9492     .        0.3000     .    
x2        0.5355    0.8445     .         .         .    
x3       -0.4407   -0.0397     .        0.7803    0.4421
x4       -0.5066    0.8622     .         .         .    
x5        1.0000     .         .         .         .    
------------------------------------------------------

We see, that $x_1$ has a component outside of that reduced space, and $x_3$ has a further component outside of the rank 3 space, and are thus partly independent of that 2-dimensional subspace (which can thus be given the term "co-planarity"). We can now decide which of the three variables $x_2,x_4$ or $x_5$ can be removed to overcome the multi-collinearity problem.

If we would use some software which does not allow this flexible reordering "inside" the rotation-parameters/procedure, we would re-order the variables forming the correlation-matrix and would do the cholesky-decomposition to arrive at something like:

[26] L1 = cholesky(...) // something in your favorite software...

L1=       f1        f2        f3        f4        f5
------------------------------------------------------
x5        1.0000     .         .         .         .   Co-planar subset  
x2        0.5355    0.8445     .         .         .    
x4       -0.5066    0.8622     .         .         .    
------------------------------------------------------ 
x1        0.0952   -0.9492     .        0.3000     .      further linearly independent
x3       -0.4407   -0.0397     .        0.7803    0.4421  variables 

[update]: Note that the candidates from which we would remove one, were not necessarily recognized by the inspection of correlations in the correlation-matrix. There the highest correlation is 0.8666 between $x_1$ and $x_4$ - but $x_1$ does not contribute to the rank-deficiency! Furthermore, the correlations between $x_2,x_4,x_5$ are all in an "acceptable" range when one wants to apply some jackknife-estimate for the removal of high-correlations assuming multicollinearity - one would not look at them as the most natural candidates from the set of bivariate correlations only.

Answer 2

Something to consider are the subtle ways your data may be collinear.

Multicollinearity: greatest hits & common mistakes:

1. Complete dummy variables (I'd guess the most common error)

Let's say your regression includes a constant and a dummy variable for January, a dummy for February, etc... all the way to December. Let's call these dummies $x_1$, $x_2$, ...., $x_{12}$.

$$ x_1 + x_2 + \ldots + x_{11} + x_{12} = 1$$ Therefore your data is collinear. If you're including dummy variables, you need to leave the dummy for one category out.

2. Components that add together to some total

I'll give a corporate finance example.

Imagine $x_1$ is short-term debt. $x_2$ is long-term debt, and $x_3$ is total debt. Obviously we have:

$$ x_3 = x_1 + x_2 $$

If you put all three on the right hand side your, your regressors are collinear.

3. Context specific linear identities

Imagine you have some regression with assets, liabilities, and shareholder's equity on the right hand side. Sound legit? Nope! The most basic accounting identity is that:

$$ \mathrm{Assets}_i = \mathrm{Liabilities}_i + \mathrm{ShareholderEquity}_i$$

Mechanically, that equation always holds true. Including two of the three would be fine. Including all three makes your data collinear.

4. Some more subtle examples...

Let $x_1$ be the age of your individual. Let $x_2$ be years of schooling.

If everyone in your sample starts school at the same age and doesn't drop out, then $x_1$ and $x_2$ are collinear.

Conclusion

Observe that in (1), (2), and (3), your pairwise correlations would be useless in identifying the collinearity problem. These are all examples where THREE or more variables are collinear, not the obvious situation where $x_1 = \alpha x_2$.

This is also why the Cholesky decomposition @Gottfried Helms discusses might be useful.

Answer 3

A correlation matrix is positive semidefinite, by definition.

However, the vast majority of correlation matrices are actually positive definite. There is a detailed explanation of this at the following link: http://www.uic.edu/classes/bstt/bstt580/jw4e/nts02.htm in sect 2.6)). This means that you have at least one redundant variable in your analysis. How many variables are you working with?

Answer 4

This can happen if you have missing values and you are computing pairwise correlation matrix instead of listwise.