# Why do we need PCA whitening before feeding into autoencoder?

In the UFLDL tutorial, we saw that autoencoder can not compress data with uncorrelated random variables. 'If the input were completely random---say, each variable comes from an IID Gaussian independent of the other features---then this compression task would be very difficult.'

But, it is suggested to apply PCA whitening before feeding the data into autoencoder. Using PCA, the data can be represented using new orthogonal variables, which are uncorrelated.

Can the autoencoder compress the data using hidden layer after PCA preprocessing?

• What is the UFLDL tutorial where "we saw" this? What is the autoencoder that you are using? What are you doing w/ it, are you predicting a known class for each pattern? Some more information would help here. Apr 7, 2015 at 1:46
• The UFLDL tutorial is here: ufldl.stanford.edu/wiki/index.php/Autoencoders_and_Sparsity. Apr 7, 2015 at 1:53

Whitening effectively changes the objective function. Let $C$ be the covariance of the inputs and $W$ be PCA whitening, \begin{align} C &= QDQ^\top, & W &= D^{-\frac{1}{2}}Q^\top. \end{align} Further, let $x$ be some input, $\hat x$ be the output of the autoencoder and $y = Wx$ be the whitened signal. Then \begin{align} ||y - \hat y||_2^2 &= ||W x - W \hat x||_2^2 \\ &= (Wx - W\hat x)^\top (Wx - W\hat x) \\ &= (x - \hat x)^\top W^\top W (x - \hat x) \\ &= (x - \hat x)^\top C^{-1} (x - \hat x) \\ &= ||x - \hat x||_{C^{-1}}^2. \end{align} That is, by optimizing $\hat y$ instead of $\hat x$, we are effectively optimizing a particular Mahalanobis distance instead of standard Euclidean distance.