Is using PCA then autoencoders for pre-processing useful? I was just reading a paper, seeing someone do the following:
As a pre-processing step they did the following:
PCA the original data -> Stacked Autoencoder
Then they fed this pre-processed data into a feed-forward neural network. But - there are two issues that I have.
Why would they PCA the data? Shouldn't the stacked autoencoder be able to learn the linear representations? 
Secondly, I noticed that they used the SAE for pre-processing. I mostly see SAEs for pre-training. That is, for initializing the weights of a feed-forward neural network. Do these "pre-processing" and "pre-training" steps cause different results? What are the implications of "pre-processing" data with the SAE to be fed into a FF instead of "pre-training" the weights of a FF with a SAE?
Thanks in advance.
 A: I'm not a deep learning expert by any means, but my guess is that the PCA serves two functions: computational improvements if the input dimensionality is significantly reduced, and a kind of preconditioning for the optimization problem. Although a normal autoencoder setup certainly can learn the linear relationships, it may make the learning process easier if that step is useful and initialized with it. Broadly speaking, it should be about equivalent to pretraining the first layer of the autoencoder with the principal components (if you don't drop much in the PCA).
Preprocessing with an autoencoder is often used to derive features to use in some other classifier or whatever. Compared to pretraining, plugging preprocessor autoencoder features into a neural net means that the final classifier can't adapt the learned features for its particular learning problem. Depending on your problem, this might hurt the performance of the final classifier somewhat. But it means that you can reuse the same learned features for multiple final classification/regression/whatever tasks, which saves a lot of training time in trying to adapt the features, and might save a substantial amount of test time if you're running a suite of learning methods on data and so can reuse the autoencoder outputs for all of them. In semi-supervised settings it might also give you better results to avoid overfitting the feature extractor to your limited labeled data.
