# Does it make sense to implement a PCA after an Autoencoder processing?

I would like to use Autoencoder to pre-process images, then extract vector from the "bottleneck" of the Autoencoder. Then, do a PCA based on the extracted vectors. I feel doing this won't make sense, because it's like repeating the same task---both Autoencoder and PCA do dimensionality reduction. However, the following is why I think I need a PCA after Autoencoder. Could someone point out where I messed up?

1. I want to reduce images with size of 224*224*3 (3 is RGB channels) to a vector with dimension 10~50. I tried through Convolution Autoencoder---if the vector extracted from the bottleneck is able to reconstruct the images, then I'm confident that the vector can represent the images.
2. I experimented a lot. However, seems vector with dimension 10~50 is not able to reconstruct the images at a good level. Then I thought may be this was because reducing dimension from 224*224*3 to 50 had lost too much information.
3. The next I did was to make the bottleneck of the Convolution Autoencoder have a size of 64*14*14 (where 64 means 64 convolution filters, 14*14 is the size of feature map). Now I'm able to well reconstruct the images.
4. But 64*14*14 has way more dimensions than expected 10~50. So I implemented a PCA based on the extracted vectors (64*14*14 dimensions). I found that the top 20 principle axes explained over 80% of the variance in the sample.

Based on the large variance explained by the 20 principle axes, can I say that the vector along the 20 principle axes is able to represent the input images?

What I'm troubled with, is that, if the top 20 principle axes explained 80+% variance in the vectors, which can be used to reconstruct images. Then why did I fail to reconstruct images with antoencoder that reduced the vector size to 20~50?

## 3 Answers

Generally, the aim of an autoencoder is to learn a representation (encoding) for a set of data, typically for the purpose of dimensionality reduction. So, the target output of the autoencoder is the autoencoder input itself.

It is shown in Auto-Association by Multilayer Perceptrons and Singular Value Decomposition that If there is one linear hidden layer and the mean squared error criterion is used to train the network, then the k hidden units learn to project the input in the span of the first k principal components of the data.

And Nonlinear Autoassociation Is Not Equivalent to PCA shows If the hidden layer is non-linear, the autoencoder behaves differently from PCA, with the ability to capture multi-modal aspects of the input distribution.

Firstly pca is a traditional method for image size reduction. Without any neural net , pca will work well for image compression. Neural nets are generally used for the more complicated problem of image recognition. I'm not sure how pca will interact with auto encoding, and I'm not convinced that there is a mathematical and general answer. I speculate as follows. PCA is essentially a linear operation, while neural nets are very non-linear. Once the non-linear operation of auto-encoding is applied, PCA will not be able to reconstruct the variables properly. In particular, it may have issues with the the third dimension since PCA is naturally a 2D tensor operator. This means it should have trouble with 3 colors.

If you are only trying to compress the images, you should follow @aginensky. But if you are using the compressed vector obtained from the autoencoder in downstream machine learning models, your PCA post-processing step can be useful, especially as some machine learning models either assume uncorrelated input data or perform better when the input data is uncorrelated. Whether or not you should retain only the top k principal components is a different matter again (if you are doing downstream machine learning), and depends a lot on how much loss of information you are willing to accept.

You could also have a look at variational autoencoders as an alternative. Here the bottleneck is much narrower, as your latent representation is not a "latent image", but a number of gaussian distributions., i.e. for each gaussian you are learning a mean and a variance. So if you'd be limiting your bottleneck to 20 distributions, you would end up with two vectors of length 20. These two vectors can be concatenated and used as input values in a non-linear machine learning model in your downstream processing.