3
$\begingroup$

What is the relationship between SVD and factor analysis? How can use singular values and other matrices from SVD to perform factor analysis or cluster document-term matrix without using other clustering techniques?

$\endgroup$
  • $\begingroup$ Welcome to CV. This question has been addressed in a number of prior threads. For instance, if you drill into the link to your question, in the right hand column are a large number of related threads. $\endgroup$ – Mike Hunter Mar 6 '16 at 18:30
  • 1
    $\begingroup$ Thanks DJohnson. I went through those questions before posting this. I couldn't find an answer to my questions from those. $\endgroup$ – dan24 Mar 6 '16 at 18:41
  • 2
    $\begingroup$ Are you sure? This one looks almost redundant with your question... stats.stackexchange.com/questions/134282/… $\endgroup$ – Mike Hunter Mar 6 '16 at 18:49
  • 1
    $\begingroup$ May be first I need to clarify the relationship between PCA and FA. From what I know, PCA and FA are different. PAC helps to combine instances into components to reduce dimension. But FA, finds related features (of the instances) and extract latent factors. Isn't it correct? Thanks $\endgroup$ – dan24 Mar 6 '16 at 18:56
  • 2
    $\begingroup$ If you are specifically about FA, not PCA. Some methods of FA have to deal with not positive definite correlation matrix on iterations; eigendecomposition can detect negative eigenvalues while svd cannot - therefore usually the former is preferred to the latter (Footnote 1). As for svd of rectangular data matrix - it is not needed and not done in FA. $\endgroup$ – ttnphns Mar 6 '16 at 19:28
1
$\begingroup$

Google brought me here, and I dislike how the comments just assume everyone knows that FA and PCA are related. So to answer your question: yes. See Tipping and Bishop, 1996. This paper is great because:

  1. It discusses the connection between FA and PCA (Section 2.2)
  2. It discusses using the SVD to compute the ML parameters (Appendix A)
| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

Google brought me here too, and I found that the implementation of Scikit-learn library, a famous repository for data science in Python, uses SVDs with a small tweak to fit the data points and perform factor analysis.

Hence the answer is a big YES you can use SVD.

If you're keen with code implementation, I suggest you can read the Factor Analysis source code of Scikit-learn here at github. They implement the SVD algorithm using Scipy library and tweak the output for shape adjustment.

In addition to that, I want to add some reference on top of Probabilistic Principal Component Analysis paper PPCA paper suggested by @gwg:

  • David Barber, Bayesian Reasoning and Machine Learning, Algorithm 21.1 (textbook) textbook here
  • Christopher M. Bishop: Pattern Recognition and Machine Learning, Chapter 12.2.4 pdf here (paper, same author of PPCA). Scikit-learn referenced this paper for their alogrithm
| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ The last link pdf is only the Contents of the book $\endgroup$ – ttnphns Jun 26 '19 at 19:06
  • $\begingroup$ Ups you're right. I gave the wrong link. I have updated the download link, but eventually you should find the free pdf of the last link through google immediately. $\endgroup$ – Daniel Kurniadi Jun 27 '19 at 10:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.