4
$\begingroup$

Following on from: What're the differences between PCA and autoencoder?

If I want to do dimensionality reduction and restrict myself to using a linear activation for my autoencoder, is there any reason to use an AE over Principal Component Analysis? PCA minimises the squared distance between our data restricted in some subspace and the true representation and has a closed form solution so is very fast - an AE really goes round the houses to achieve this and the solution will have no better loss i.e. the result of the loss function being optimised could be at best the same as for PCA.

So I conclude that I should only ever use an autoencoder for dimensionality reduction if I want to incorporate a non-linear activation function, or use multiple layers, or use a convolutional layer etc. Is this a fair conclusion?

$\endgroup$
  • $\begingroup$ Sorry, this is a typo. Should read no better loss. By 'no better loss' I mean the result of the loss function, e.g. least squares distance, will be no smaller than that of PCA. $\endgroup$ – James Owers Mar 31 '16 at 11:05
  • 2
    $\begingroup$ I think your understanding is correct & your conclusions are fair. I am not sure what to add to that, so I am not posting this as an answer :-) $\endgroup$ – amoeba Mar 31 '16 at 11:35
8
$\begingroup$

Using a linear autoencoder instead of PCA could also be useful in a large-scale learning scenario. Since you can use Stochastic Gradient Descent (SGD) to train the AE, there is no neeed to load all the training samples in the main memory at once, which can be problematic with large-scale problems.

The linear AE may also come handy in online-learning scenarios, where features arrive over time, as this could be easily handled with SGD.

Another option would be using an Incremental version of PCA (e.g., that of Scikit-learn): http://scikit-learn.org/stable/auto_examples/decomposition/plot_incremental_pca.html

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.