Both PCA and autoencoder can do demension reduction, so what are the difference between them? In what situation I should use one over another?

up vote 42 down vote accepted

PCA is restricted to a linear map, while auto encoders can have nonlinear enoder/decoders.

A single layer auto encoder with linear transfer function is nearly equivalent to PCA, where nearly means that the $W$ found by AE and PCA won't be the same--but the subspace spanned by the respective $W$'s will.

  • I see! So i need to have two layers for non-linear transformation. So multiple layers means very complex non-linear? – RockTheStar Oct 15 '14 at 16:49
  • 5
    @RockTheStar: it's not the number of layers that matters, but the activation function [transfer function]. With linear transfer function, no number of layers will lead to a non-linear autoencoder. – amoeba Oct 15 '14 at 21:13
  • So, with non-linear transformation, even there is only 1 layer of hidden unit. The solution is still non-linear? – RockTheStar Oct 15 '14 at 22:40
  • Yes. (Also it might still be linear in some cases, e.g. when the hidden units are activated in near linear regions.) – bayerj Oct 16 '14 at 6:07
  • "when the hidden units are activated in the near linear regions", you mean the linear part in the sigmoid function, right? – RockTheStar Oct 16 '14 at 17:26

As bayerj points out PCA is method that assumes linear systems where as Autoencoders (AE) do not. If no non-linear function is used in the AE and the number of neurons in the hidden layer is of smaller dimension then that of the input then PCA and AE can yield the same result. Otherwise the AE may find a different subspace.

One thing to note is that the hidden layer in an AE can be of greater dimensionality than that of the input. In such cases AE's may not be doing dimensionality reduction. In this case we perceive them as doing a transformation from one feature space to another wherein the data in the new feature space disentangles factors of variation.

Regarding to your question about whether multiple layers means very complex non-linear in your response to bayerj. Depending on what you mean by "very complex non-linear" this could be true. However depth is really offering better generalization. Many methods require an equal number of samples equal to the number of regions. However it turns out that "a very large number of regions, e.g., $O(2^N)$, can be defined with $O(N)$ examples" according to Bengio et al. This is a result of the complexity in representation that arises from composing lower features from lower layers in the network.

  • 1
    thx for your ans! – RockTheStar Oct 16 '14 at 17:27

This is better suited as a comment but as I lack the reputation for that it will be given as an answer.

I somewhat confused with notion of nearly in bayerj:s answer. Reading Neural Networks and Principal Component Analysis: Learning from Examples Without Local Minima where the proof is given.

''In the auto-associative case ... and therefore the unique locally and globally optimal map W is the orthogonal projection onto the space spanned by the first $p$ eigenvectors of $\Sigma_{XX}$''

Is this then not the exactly the correspondent space as spanned by PCA?

  • The paper you cite uses a linear autoencoder, i.e., no non-linear activation function. That is why its weights span the same subspace spanned by PCA exactly. – elliotp Feb 22 at 11:45

The currently accepted answer by @bayerj states that the weights of a linear autoencoder span the same subspace as the principal components found by PCA, but they are not the same vectors. In particular, they are not an orthogonal basis. This is true, however we can easily recover the principal components loading vectors from the autoencoder weights. A little bit of notation: let $\{\mathbf{x}_i \in \mathbb{R}^n \}_{i=1}^N $ be a set of $N$ $n-$dimensional vectors, for which we wish to compute the PCA, and let $X$ be the matrix whose columns are $\mathbf{x}_1,\dots,\mathbf{x}_N$. Then, let's define a linear autoencoder as the one-hidden layer neural network defined by the following equations:

$$ \begin{align} \mathbf{h}_1 & = \mathbf{W}_1\mathbf{x} + \mathbf{b}_1 \\ \hat{\mathbf{x}} & = \mathbf{W}_2\mathbf{h}_1 + \mathbf{b}_2 \end{align} $$

where $\hat{\mathbf{x}}$ is the output of the (linear) autoencoder, denoted with a hat in order to stress the fact that the output of an autoencoder is a "reconstruction" of the input. Note that, as it's most common with autoencoders, the hidden layer has less units than the input layer, i.e., $W_1\in \mathbb{R}^{n \times m}$ and $W_2\in \mathbb{R}^{m \times n}$ with $m < n$.

Now, after training your linear autoencoder, compute the first $m$ singular vectors of $W_2$. It's possible to prove that these singular vectors are actually the first $m$ principal components of $X$, and the proof is in Plaut, E.,From Principal Subspaces to Principal Components with Linear Autoencoders, Arxiv.org:1804.10253.

Since SVD is actually the algorithm commonly used to compute PCA, it could seem meaningless to first train a linear autoencoder and then apply SVD to $W_2$ in order to recover then first $m$ loading vectors, rather than directly applying SVD to $X$. The point is that $X$ is a $n \times N$ matrix, while a $W_2$ is $m\times n$. Now, the time complexity of SVD for $W_2$ is $O(m^2n)$, while for $X$ is $O(n^2N)$ with $m < n $, thus some saving could be attained (even if not as big as claimed by the author of the paper I link). Of course, there are other more useful approaches to compute the PCA of Big Data (randomized online PCA comes to mind), but the main point of this equivalence between linear autoencoders and PCA is not to find a practical way to compute PCA for huge data sets: it's more about giving us an intuition on the connections between autoencoders and other statistical approaches to dimension reduction.

protected by kjetil b halvorsen Apr 3 at 20:36

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.