Following on from: What're the differences between PCA and autoencoder?

If I want to do dimensionality reduction and restrict myself to using a linear activation for my autoencoder, is there any reason to use an AE over Principal Component Analysis? PCA minimises the squared distance between our data restricted in some subspace and the true representation and has a closed form solution so is very fast - an AE really goes round the houses to achieve this and the solution will have no better loss i.e. the result of the loss function being optimised could be at best the same as for PCA.

So I conclude that I should only ever use an autoencoder for dimensionality reduction if I want to incorporate a non-linear activation function, or use multiple layers, or use a convolutional layer etc. Is this a fair conclusion?

  • $\begingroup$ Sorry, this is a typo. Should read no better loss. By 'no better loss' I mean the result of the loss function, e.g. least squares distance, will be no smaller than that of PCA. $\endgroup$ Commented Mar 31, 2016 at 11:05
  • 2
    $\begingroup$ I think your understanding is correct & your conclusions are fair. I am not sure what to add to that, so I am not posting this as an answer :-) $\endgroup$
    – amoeba
    Commented Mar 31, 2016 at 11:35

1 Answer 1


Using a linear autoencoder instead of PCA could also be useful in a large-scale learning scenario. Since you can use Stochastic Gradient Descent (SGD) to train the AE, there is no neeed to load all the training samples in the main memory at once, which can be problematic with large-scale problems.

The linear AE may also come handy in online-learning scenarios, where the training examples arrive over time, as this could be easily handled with SGD.

Another option would be using an Incremental version of PCA (e.g., that of Scikit-learn): http://scikit-learn.org/stable/auto_examples/decomposition/plot_incremental_pca.html


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.