# Why do deep learning practitioners forego PCA for ZCA?

I have an understanding of PCA and ZCA, read a similar question on the subject which, unfortunately, does not have the specific answer to my question.

I understand the benefits of data whitening: specifically, standardizing the dynamic range of each data feature, which is very important when using stochastic gradient descent. What I fail to understand is opting to use ZCA and foregoing the benefit of having de-correlate your features.

I understand that it is more appealing to the human eye, but aren't we making the job of generalizing the data harder for the learning algorithm?

• "foregoing the benefit of having de-correlate your features" -- what do you mean? ZCA de-corelates the features. May 16 '16 at 9:24
• Regarding the benefits of ZCA over PCA for deep learning, have you read the last paragraph of my answer in the linked question? May 16 '16 at 9:26
• What is ZCA, what is SDG? can you explain (as an edit to the post). May 16 '16 at 14:21
• Thanks, but posts should be selfexplaing so please add this information to the post! May 16 '16 at 15:14
• Actually, "standardizing the dynamic range of each data feature" is NOT the reason to whiten; if the goal is to standardize each feature one can simply standardize each feature. Whitening does much more. May 16 '16 at 16:35

A big benefit for ZCA is that whitened data is still a picture in the same space as the original. If you ZCA whiten a photo of a cat, it's still cat-like. This is helpful for other techniques searching for nonlinear structure. You're able to take an $n\times n$ patch from a picture and apply a filter to it with the belief that the pixels will exhibit certain useful dependencies by virtue of being neighbours. E.g. is there an eye in this patch? Is there fur in this patch? The same is emphatically not true of PCA, which completely disregards the spatial structure of images.
Second, contrary to your statement, ZCA does decorrelate data. The development in Bell 1997 - equations 5 and 8 - makes this a requirement of the technique. Take the covariance matrix $\bf \Sigma$ and use eigendecomposition to form the whitening matrix $\bf W_z = U D^{-1/2} U^T$. Then for some new $\bf x$ drawn from the distribution we have $Cov(\bf W_zx, \bf W_zx $$)=\bf W_z$$Cov(\bf x,x$)$\bf W_z^T= I$.
• Take an vector in $R^N$ and ZCA whiten it. The result is a vector in $R^N$. The same is not true for PCA. While you could find vectors in $R^N$ that make bad pictures, you're going to have even more trouble if you can't even look at the vector as a picture. May 16 '16 at 9:39
• @user2324712, to your line perform another rotation on the data after PCA whitening, the eigen-vectors shouldn't be be orthogonal anymore. Although it's not very clear to me what you call "eigenvectors" after additional orthogonal rotation, - let me recommend an answer with a chart, about rotation after PCA. "Standardized PC scores" is a synonym to "PCA-whitened data". One should not mix up, when speaking of orthogonality, rotated PC data and corresponding rotated loadings. May 16 '16 at 10:36