- PCA is recommended before AE for uncorrelating the inputs (ref).
- Autoencoders being neural networks, they're good for non-linear dimensionality reduction (wikipedia).
- PCA is a linear technique
With PCA ahead of AE, doesn't this screw up the ability of the AE to detect non-linear relationships?
EDIT 1: I ran a few experiments here for using PCA vs PCA+AE in the case of linear dependencies
My generated input data consists of 3 main signals, along with duplicates of those 3 main signals
- sine wave with frequency = 1 (call this sin1)
- sine wave with frequency = 2 (call this sin2)
- sine wave = sin1 + sin2
- If you scroll down to "AE without PCA", it shows that the encoded 2 features are not perfectly sinusoidal.
- Further down, "PCA without dimensionality reduction" shows that it detects that there are mainly 2 sine waves (sin1 and sin2), but "sin1" is kind of slurred
- Finally, "AE with PCA (and using dimensionality reduction for PCA) shows that AE does recover the "sin1" without the "slur", but now the "sin2" is sort of more "sawtooth" than before.
So far, I see that AE after PCA worked better than PCA alone.
Any ideas on how I can modify my experiment to demonstrate if this fails?
EDIT 2: I ran a few experiments here for using PCA vs PCA+AE in the case of non-linear dependencies
This experiment is close to the one before, but instead of the 3rd sine wave = sin1 + sin2, I use sin1*sin2.
This creates a non-linear dependency in the input columns. A few things to notice here
- PCA now detects 3 main components
- AE without PCA, with encoding to 3 signals, encodes non-sinusoidals
- AE without PCA, with encoding to 2 signals, encodes to something close to a sinusoidal
- PCA reducing to 3 + AE keeping at 3, encodes to non-sinusoidals (similar to AE without PCA)
- PCA reducing to 3 + AE reducing to 2, encodes to something close to sinusoidal (similar to AE without PCA)
I'd like to conclude that in this case, PCA+AE is as good as AE alone. Does my conclusion make sense? Or am I mis-interpreting the results?