I am experimenting a bit autoencoders, and with tensorflow I created a model that tries to reconstruct the MNIST dataset.
My network is very simple: X, e1, e2, d1, Y, where e1 and e2 are encoding layers, d2 and Y are decoding layers (and Y is the reconstructed output).
X has 784 units, e1 has 100, e2 has 50, d1 has 100 again and Y 784 again.
I am using sigmoids as activation functions for layers e1, e2, d1 and Y. Inputs are in [0,1] and so should be the outputs.
Well, I tried using cross entropy as loss function, but the output was always a blob, and I noticed that the weights from X to e1 would always converge to an zero-valued matrix.
On the other hand, using mean squared errors as loss function, would produce a decent result, and I am now able to reconstruct the inputs.
Why is that so? I thought I could interpret the values as probabilities, and therefore use cross entropy, but obviously I am doing something wrong.