5
$\begingroup$

I was just reading a paper, seeing someone do the following:

As a pre-processing step they did the following:

PCA the original data -> Stacked Autoencoder

Then they fed this pre-processed data into a feed-forward neural network. But - there are two issues that I have.

Why would they PCA the data? Shouldn't the stacked autoencoder be able to learn the linear representations?

Secondly, I noticed that they used the SAE for pre-processing. I mostly see SAEs for pre-training. That is, for initializing the weights of a feed-forward neural network. Do these "pre-processing" and "pre-training" steps cause different results? What are the implications of "pre-processing" data with the SAE to be fed into a FF instead of "pre-training" the weights of a FF with a SAE?

Thanks in advance.

$\endgroup$
1
$\begingroup$

I'm not a deep learning expert by any means, but my guess is that the PCA serves two functions: computational improvements if the input dimensionality is significantly reduced, and a kind of preconditioning for the optimization problem. Although a normal autoencoder setup certainly can learn the linear relationships, it may make the learning process easier if that step is useful and initialized with it. Broadly speaking, it should be about equivalent to pretraining the first layer of the autoencoder with the principal components (if you don't drop much in the PCA).

Preprocessing with an autoencoder is often used to derive features to use in some other classifier or whatever. Compared to pretraining, plugging preprocessor autoencoder features into a neural net means that the final classifier can't adapt the learned features for its particular learning problem. Depending on your problem, this might hurt the performance of the final classifier somewhat. But it means that you can reuse the same learned features for multiple final classification/regression/whatever tasks, which saves a lot of training time in trying to adapt the features, and might save a substantial amount of test time if you're running a suite of learning methods on data and so can reuse the autoencoder outputs for all of them. In semi-supervised settings it might also give you better results to avoid overfitting the feature extractor to your limited labeled data.

$\endgroup$
  • $\begingroup$ Hmm interesting. So basically this concept of "pre-training" the weights (proposed by Hinton) of a FFnet using an autoencoder and "pre-processing" using an autoencoder are basically the same? $\endgroup$ – Steven Dec 27 '14 at 16:03
  • $\begingroup$ Well, they're related. The difference is that if you pre-train, you can then adapt the results of the autoencoder for your final learning problem, where if they're pre-processed, you stick with whatever you get out (which is both good and bad). PCA is different because it's linear, so (if you don't drop many components) the net can basically just reverse it if it decides it needs to (though it'd be a little hard to learn). $\endgroup$ – Dougal Dec 29 '14 at 16:38
  • $\begingroup$ One issue is that, with PCA - you are hoping that the PCs that you don't keep are noise. Obviously, this can be an issue. But it seems that it simplifies the training procedure if we PCA before hand. In fact, I've seen Andrew Ng do it in one of his papers. $\endgroup$ – Steven Dec 31 '14 at 15:40
  • $\begingroup$ But I suppose my question is: how exactly difficult (on average) would it be for an autoencoder to learn the PCA algorithm? If PCA can do it, I think an autoencoder could do it as well. $\endgroup$ – Steven Dec 31 '14 at 15:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.