Building an autoencoder in Tensorflow to surpass PCA Hinton and Salakhutdinov in Reducing the Dimensionality of Data with Neural Networks, Science 2006 proposed a non-linear PCA through the use of a deep autoencoder. I have tried to build and train a PCA autoencoder with Tensorflow several times but I have never been able to obtain better result than linear PCA. 
How can I efficiently train an autoencoder?
(Later edit by @amoeba: the original version of this question contained Python Tensorflow code that did not work correctly. One can find it in the edit history.)
 A: Here is my jupyter notebook where I try to replicate your result, with the following differences:


*

*instead of using tensorflow directly, I use it view keras

*leaky relu instead of relu to avoid saturation (i.e. encoded output being 0)


*

*this might be a reason for poor performance of AE


*autoencoder input is data scaled to [0,1]


*

*I think I read somewhere that autoencoders with relu work best with [0-1] data

*running my notebook with autoencoders' input being the mean=0, std=1 gave MSE for AE > 0.7 for all dimensionality reductions, so maybe this is one of your problems



*PCA input is kept being data with mean=0 and std=1


*

*This may also mean that the MSE result of PCA is not comparable to the MSE result of PCA

*Maybe I'll just re-run this later with [0-1] data for both PCA and AE



*PCA input is also scaled to [0-1]. PCA works with (mean=0,std=1) data too, but the MSE would be incomparable to AE


My MSE results for PCA from dimensionality reduction of 1 to 6
(where the input has 6 columns)
and for AE from dim. red. of 1 to 6 are below:

With PCA input being (mean=0,std=1) while AE input being [0-1] range
- 4e-15 : PCA6
- .015  : PCA5
- .0502 :        AE5
- .0508 :        AE6
- .051  :        AE4
- .053  :        AE3
- .157  : PCA4
- .258  :        AE2
- .259  : PCA3
- .377  :        AE1
- .483  : PCA2
- .682  : PCA1



*

*9e-15 : PCA6

*.0094 : PCA5

*.0502 : AE5

*.0507 : AE6

*.0514 : AE4

*.0532 : AE3

*.0772 : PCA4

*.1231 : PCA3

*.2588 : AE2

*.2831 : PCA2

*.3773 : AE1

*.3885 : PCA1


Linear PCA with no dimensionality reduction can achieve 9e-15 because it can just push whatever it was unable to fit into the last component.
