# Factor analysis problem — singular covariance matrix?

I'm having trouble performing factor analysis on my dataset.

When I perform the factor analysis in SPSS (default settings), it works fine. Problem is, I need to do it programmatically (in Python). When I try using Python (MDP library) to do factor analysis on the same dataset, I get this error:

"The covariance matrix of the data is singular. Redundant dimensions need to be removed"

Upon looking into the MDP documentation, it says "...returns the Maximum A Posteriori estimate of the latent variables." Being a factor analysis newbie, I wasn't too clear on what this meant, but I tried changing the default extraction method in SPSS from "principal components" to "maximum likelihood". Then, in SPSS, I get the error:

"This matrix is not positive definite."

Are these two errors the same thing? Regardless, what can I do to fix my dataset so that the covariance matrix is not singular?

Thanks!

edit: OK, so I was trying to keep things simplified, but perhaps its better to just explain everything from the start.

I have a series of documents. Yes, I'm only using 9 documents as a simple test case, but my final objective will be to use it on a much larger corpus.

I've built a term-document matrix, performed tf-idf, and did SVD-- mostly with the help of blog.josephwilk.net/.../latent-semantic-analysis-in-python.html

Now I have a reconstructed matrix, and I want to sort the documents into categories. So, I tried using factor analysis. In fact, it seems to work-- when I put it in SPSS, the factor loadings indicate that the documents are grouped the way I thought they should be, and the loading are higher than if I hadn't performed SVD. (Although I think technically, SPSS is doing PCA even though its under the 'Factor Analysis' heading).

I tried using MDP's PCANode, but that doesn't seem to give me anything close to what I want. Strangely, if I transpose my matrix, the factor analysis does work (it will group the terms, instead of the documents).

Hopefully this all makes a little more sense now...

• What do you mean by the "reconstructed matrix"? If you do the SVD, keep a few components, and then do PCA, the results should be identical. – JMS May 23 '11 at 1:14
• The LSA code I was using (from joseph wilk's blog, in my post) "reconstructs" the matrix after SVD. To be honest, I don't really know much about SVD or what it's doing. I'm still having some problems, but I think we've moved beyond my original question so I'll marked this solved. Thanks! – Jeff May 24 '11 at 21:25
• I imagine what it's doing is "reconstructing" the matrix from the first few singular vectors. The link to the blog post is broken btw. But that would make absolutely no difference if you go on and do PCA to the reconstructed matrix! Perhaps it's centering/scaling for you, then you would get different results. I'd look into it further; it seems like you want to be working with the scores from LSA to do clustering/categorization/etc – JMS May 25 '11 at 3:44
• Link works for me, but I think I'm going to explore gensim further and see if that pans out. Thanks again! – Jeff May 25 '11 at 3:48
• Weird - link isn't a link at all for me. Gensim is a pretty mature piece of kit, you should do well with it. If you have to process and corpora the natual language toolkit ( aka NLTK) has some useful bits & is super well documented, although it's more NLP oriented. Good luck :) – JMS May 25 '11 at 3:55

Yes, the two errors amount to the same thing. They're telling you (roughly) that two or more of your manifest variables are linearly dependent (like $y_1 = ay_2 + b$ for scalars $a, b$). These two variables (dimensions) would be "redundant", meaning that the sample covariance matrix is not invertible (ie is singular) and therefore not positive definite either.