I have an understanding of PCA and ZCA, read a similar question on the subject which, unfortunately, does not have the specific answer to my question.

I understand the benefits of data whitening: specifically, standardizing the dynamic range of each data feature, which is very important when using stochastic gradient descent. What I fail to understand is opting to use ZCA and foregoing the benefit of having de-correlate your features.

I understand that it is more appealing to the human eye, but aren't we making the job of generalizing the data harder for the learning algorithm?

  • $\begingroup$ "foregoing the benefit of having de-correlate your features" -- what do you mean? ZCA de-corelates the features. $\endgroup$
    – amoeba
    May 16 '16 at 9:24
  • $\begingroup$ Regarding the benefits of ZCA over PCA for deep learning, have you read the last paragraph of my answer in the linked question? $\endgroup$
    – amoeba
    May 16 '16 at 9:26
  • $\begingroup$ What is ZCA, what is SDG? can you explain (as an edit to the post). $\endgroup$ May 16 '16 at 14:21
  • 1
    $\begingroup$ Thanks, but posts should be selfexplaing so please add this information to the post! $\endgroup$ May 16 '16 at 15:14
  • 3
    $\begingroup$ Actually, "standardizing the dynamic range of each data feature" is NOT the reason to whiten; if the goal is to standardize each feature one can simply standardize each feature. Whitening does much more. $\endgroup$
    – amoeba
    May 16 '16 at 16:35

A big benefit for ZCA is that whitened data is still a picture in the same space as the original. If you ZCA whiten a photo of a cat, it's still cat-like. This is helpful for other techniques searching for nonlinear structure. You're able to take an $n\times n$ patch from a picture and apply a filter to it with the belief that the pixels will exhibit certain useful dependencies by virtue of being neighbours. E.g. is there an eye in this patch? Is there fur in this patch? The same is emphatically not true of PCA, which completely disregards the spatial structure of images.

Second, contrary to your statement, ZCA does decorrelate data. The development in Bell 1997 - equations 5 and 8 - makes this a requirement of the technique. Take the covariance matrix $\bf \Sigma$ and use eigendecomposition to form the whitening matrix $\bf W_z = U D^{-1/2} U^T$. Then for some new $\bf x$ drawn from the distribution we have $Cov(\bf W_zx, \bf W_zx $$)=\bf W_z$$Cov(\bf x,x$)$\bf W_z^T= I$.

  • $\begingroup$ I agree with your answer from 2nd sentence onwards, but I don't understand the 1st one: what does "in the same space as the original" mean? $\endgroup$
    – amoeba
    May 16 '16 at 9:35
  • $\begingroup$ Take an vector in $R^N$ and ZCA whiten it. The result is a vector in $R^N$. The same is not true for PCA. While you could find vectors in $R^N$ that make bad pictures, you're going to have even more trouble if you can't even look at the vector as a picture. $\endgroup$ May 16 '16 at 9:39
  • 1
    $\begingroup$ 1) You're making an assumption about what would or wouldn't be good in practice. Deep learners are reputed to try everything under the sun to see if it works. I put it to you that if PCA worked better in practice, they'd be doing PCA. $\endgroup$ May 16 '16 at 10:29
  • 2
    $\begingroup$ 2) You're also assuming that separation by variance is the best way to actually separate data. If you consider the performance of support vector machines with nonlinear kernels versus linear ones, you'll see this is not a safe assumption to make. $\endgroup$ May 16 '16 at 10:34
  • 2
    $\begingroup$ @user2324712, to your line perform another rotation on the data after PCA whitening, the eigen-vectors shouldn't be be orthogonal anymore. Although it's not very clear to me what you call "eigenvectors" after additional orthogonal rotation, - let me recommend an answer with a chart, about rotation after PCA. "Standardized PC scores" is a synonym to "PCA-whitened data". One should not mix up, when speaking of orthogonality, rotated PC data and corresponding rotated loadings. $\endgroup$
    – ttnphns
    May 16 '16 at 10:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.