Factor Analysis vs Autoencoders

Question

1. What is the difference between factor analysis and autoencoders.
2. When should we use one and not the other?

Answer 1

Simply saying, linear factor analysis methods like PCA give you projections to a low-dimensional hyperplane (like line in 2D space). So, the only dependency PCA could find is just correlation/covariance. You might read more about other methods on Wikipedia - regardless of approach, they usually exploit the information obtained from covariance matrix, PCA (and its extensions like varimax) is just one way to do that. The main advantage of those methods is that they usually solve some tractable or even closed-form problem.

For instance, PCA is solving problem of finding the roots of a polynomial; so, finding eigen-decomposition requires iterative algorithm with complexity around: $$O(n^3+n^2\log^2 n\log b),$$ which is still "faster" than "unpredictable" convergence of neural networks used for auto-encoders. So, those "linear" algorithms are simpler and easier to debug and interpret. Some of them, like SFA, Sparse coding, ICA actually use gradient descend, but are simpler to debug than AE as it's easier to understand results.

Source: http://www.slideshare.net/zukun/icml2012-tutorial-representationlearning

Auto-encoders, on the other hand, can learn a "manifold" (like curve in 2D-space), so it might find more complicated "non-linear" factors, or might not if there is not enough network's capacity - then you'll get same result as PCA at best. Note, that auto-encoders are basically just feed-forward MLP (sometimes even with just one hidden layer per encoder and one for decoder), so you need to have a proper regularization; otherwise, AE might just learn an identity function. It's usually harder to interpret results of AE as it outputs data laying on some "hyper-surface", that is often seen as a manifold by ML-researchers. Basically that's why neural networks are involved here - in order to catch those non-linearities. AE usually require some iterative method (like SGD) to solve optimization problem - and they might require much more computational resources than linear methods.

You can also read about manifold learning here: Nonlinear dimensionality reduction

Example of denoising auto-encoder: Source: http://www.slideshare.net/zukun/icml2012-tutorial-representationlearning

---

How to understand results:

If you use eigen-decomposition, the PCA algorithm will return you $Q$ and $\Lambda$, so you can take vectors with biggest corresponding eigenvalues in order to reduce dimensionality. Auto-encoder will return you a matrix with columns corresponding to the reduced vectors, so you have to know the number of factors in advance, or try to make the resulting matrix more sparse.

The practical recommendation is to try an appropriate linear method first (it should work faster for reasonable amount of features). If it's not representative enough - try some auto-encoders.

---

Conclusion

TLDR: you can consider auto-encoders as a more general case (non-linear version) of linear factor analysis.

P.S. You might find more complete answer in "deep learning book" - they describe both approaches (vanilla and state-of-the-art)