• PCA is recommended before AE for uncorrelating the inputs (ref).
  • Autoencoders being neural networks, they're good for non-linear dimensionality reduction (wikipedia).
  • PCA is a linear technique

With PCA ahead of AE, doesn't this screw up the ability of the AE to detect non-linear relationships?

EDIT 1: I ran a few experiments here for using PCA vs PCA+AE in the case of linear dependencies

My generated input data consists of 3 main signals, along with duplicates of those 3 main signals

  1. sine wave with frequency = 1 (call this sin1)
  2. sine wave with frequency = 2 (call this sin2)
  3. sine wave = sin1 + sin2

My observations:

  1. If you scroll down to "AE without PCA", it shows that the encoded 2 features are not perfectly sinusoidal.
  2. Further down, "PCA without dimensionality reduction" shows that it detects that there are mainly 2 sine waves (sin1 and sin2), but "sin1" is kind of slurred
  3. Finally, "AE with PCA (and using dimensionality reduction for PCA) shows that AE does recover the "sin1" without the "slur", but now the "sin2" is sort of more "sawtooth" than before.

So far, I see that AE after PCA worked better than PCA alone. Any ideas on how I can modify my experiment to demonstrate if this fails?

EDIT 2: I ran a few experiments here for using PCA vs PCA+AE in the case of non-linear dependencies

This experiment is close to the one before, but instead of the 3rd sine wave = sin1 + sin2, I use sin1*sin2.

This creates a non-linear dependency in the input columns. A few things to notice here

  1. PCA now detects 3 main components
  2. AE without PCA, with encoding to 3 signals, encodes non-sinusoidals
  3. AE without PCA, with encoding to 2 signals, encodes to something close to a sinusoidal
  4. PCA reducing to 3 + AE keeping at 3, encodes to non-sinusoidals (similar to AE without PCA)
  5. PCA reducing to 3 + AE reducing to 2, encodes to something close to sinusoidal (similar to AE without PCA)

I'd like to conclude that in this case, PCA+AE is as good as AE alone. Does my conclusion make sense? Or am I mis-interpreting the results?

  • 1
    $\begingroup$ I think you're mixing up correlation and nonlinearity. The resulting PCA representation can be thought of as a vector $(v_1,...,v_n)$ where each $v_i$ represents the projection of the image on the $i$'th principle component. There's nothing preventing the autoencoder from picking up nonlinear relationships across components. $\endgroup$ – Alex R. Jul 18 '17 at 17:38
  • $\begingroup$ Hey @AlexR. I added a link to a jupyter notebook showing some experimentation I did. So far, it seems to confirm what you said. Any ideas I can try that can dis-prove your statement? (kind of devil's advocate here) $\endgroup$ – Shadi Jul 19 '17 at 16:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.