# Factor Analysis vs Autoencoders

1. What is the difference between factor analysis and autoencoders.
2. When should we use one and not the other?
• Wikipedia on Autoencoder says If linear activations are used, or only a single sigmoid hidden layer, then the optimal solution to an autoencoder is strongly related to principal component analysis. – ttnphns Feb 11 '17 at 9:30

Simply saying, linear factor analysis methods like PCA give you projections to a low-dimensional hyperplane (like line in 2D space). So, the only dependency PCA could find is just correlation/covariance. You might read more about other methods on Wikipedia - regardless of approach, they usually exploit the information obtained from covariance matrix, PCA (and its extensions like varimax) is just one way to do that. The main advantage of those methods is that they usually solve some tractable or even closed-form problem.

For instance, PCA is solving problem of finding the roots of a polynomial; so, finding eigen-decomposition requires iterative algorithm with complexity around: $$O(n^3+n^2\log^2 n\log b),$$ which is still "faster" than "unpredictable" convergence of neural networks used for auto-encoders. So, those "linear" algorithms are simpler and easier to debug and interpret. Some of them, like SFA, Sparse coding, ICA actually use gradient descend, but are simpler to debug than AE as it's easier to understand results.

Auto-encoders, on the other hand, can learn a "manifold" (like curve in 2D-space), so it might find more complicated "non-linear" factors, or might not if there is not enough network's capacity - then you'll get same result as PCA at best. Note, that auto-encoders are basically just feed-forward MLP (sometimes even with just one hidden layer per encoder and one for decoder), so you need to have a proper regularization; otherwise, AE might just learn an identity function. It's usually harder to interpret results of AE as it outputs data laying on some "hyper-surface", that is often seen as a manifold by ML-researchers. Basically that's why neural networks are involved here - in order to catch those non-linearities. AE usually require some iterative method (like SGD) to solve optimization problem - and they might require much more computational resources than linear methods.

You can also read about manifold learning here: Nonlinear dimensionality reduction

Example of denoising auto-encoder: Source: http://www.slideshare.net/zukun/icml2012-tutorial-representationlearning

How to understand results:

If you use eigen-decomposition, the PCA algorithm will return you $Q$ and $\Lambda$, so you can take vectors with biggest corresponding eigenvalues in order to reduce dimensionality. Auto-encoder will return you a matrix with columns corresponding to the reduced vectors, so you have to know the number of factors in advance, or try to make the resulting matrix more sparse.

The practical recommendation is to try an appropriate linear method first (it should work faster for reasonable amount of features). If it's not representative enough - try some auto-encoders.

Exploratory vs Confirmatory analysis

All models above (linear and "non-linear") are representing exploratory factor analysis, which means that no prior knowledge was given to the algorithm - it learned it unsupervised way. On the contrary, confirmatory analysis requires some prior knowledge/hypotesis about distribution (it's not exactly same but kind of like Bayesian approach to statistics) - so we can't directly compare it to unsupervised learning approach like AE.

However, hypothetically, there are several ways you could express prior knowledge to auto-encoder: you can initialize weights from another model, you can modify target samples (for denoising auto-encoders), you can restrict number of layers and "neurons" in the network. So finally, you can estimate such model by comparing AE's output to some validation set. Unfortunately, I can't point you to any research on that topic.

Conclusion

TLDR: you can consider auto-encoders as a more general case (non-linear version) of linear factor analysis.

P.S. You might find more complete answer in "deep learning book" - they describe both approaches (vanilla and state-of-the-art)

• well i read the wikipedia article and it says that there is a lot of controversy regarding the difference between pca and factor analysis and that pca is just basic exploratory factor analysis so what are the other aspects of factor analysis and what is the relation between the features we get from autoencoders and those we get from factor analysis are these interpretable or do they just capture some trend in terms of numerical values – arrhhh Feb 11 '17 at 10:19
• @arrhhh I've added some updates to my answer – dk14 Feb 11 '17 at 12:15
• arrhhhh yes, there is controversy. Many things have been described as Factor Analysis. @dk14 has done a pretty good job of summarising the main difference: FA fits a linear manifold to the data, autoencoders fit a nonlinear manifold. This is a fairly big conceptual difference. The difference between FA and PCA lies in what flavour of linear manifold you get. This depends on whose versions you follow. – conjectures Jul 11 '17 at 11:00