I am experimenting a bit autoencoders, and with tensorflow I created a model that tries to reconstruct the MNIST dataset.

My network is very simple: X, e1, e2, d1, Y, where e1 and e2 are encoding layers, d2 and Y are decoding layers (and Y is the reconstructed output).

X has 784 units, e1 has 100, e2 has 50, d1 has 100 again and Y 784 again.

I am using sigmoids as activation functions for layers e1, e2, d1 and Y. Inputs are in [0,1] and so should be the outputs.

Well, I tried using cross entropy as loss function, but the output was always a blob, and I noticed that the weights from X to e1 would always converge to an zero-valued matrix.

On the other hand, using mean squared errors as loss function, would produce a decent result, and I am now able to reconstruct the inputs.

Why is that so? I thought I could interpret the values as probabilities, and therefore use cross entropy, but obviously I am doing something wrong.

  • 1
    $\begingroup$ You mentioned using sigmoid for the activation functions in the first layers. Those exhibit "saturation," as explained more fully in the CS231n Notes. Have you tried another activation function like ReLUs? $\endgroup$ – Jerry Dec 1 '17 at 14:12
  • $\begingroup$ I can't recall right now if I tried other, but thanks for sharing! $\endgroup$ – AkiRoss Dec 3 '17 at 21:14
  • $\begingroup$ I think the specific problem here pertains to using saturating non-linearities. But these related questions ask about using BCE loss for non-binary targets: stats.stackexchange.com/questions/370179/… and stats.stackexchange.com/questions/394582/… $\endgroup$ – Sycorax Jul 31 at 0:25

I think the best answer to this is that the cross-entropy loss function is just not well-suited to this particular task.

In taking this approach, you are essentially saying the true MNIST data is binary, and your pixel intensities represent the probability that each pixel is 'on.' But we know this is not actually the case. The incorrectness of this implicit assumption is then causing us issues.

We can also look at the cost function and see why it might be inappropriate. Let's say our target pixel value is 0.8. If we plot the MSE loss, and the cross-entropy loss $- [ (\text{target}) \log (\text{prediction}) + (1 - \text{target}) \log (1 - \text{prediction}) ]$ (normalising this so that it's minimum is at zero), we get:

cross-entropy vs. mse loss

We can see that the cross-entropy loss is asymmetric. Why would we want this? Is it really worse to predict 0.9 for this 0.8 pixel than it is to predict 0.7? I would say it's maybe better, if anything.

We could probably go into more detail and figure out why this leads to the specific blobs that you are seeing. I'd hazard a guess that it is because pixel intensities are above 0.5 on average in the region where you are seeing the blob. But in general this is a case of the implicit modelling assumptions you have made being inappropriate for the data.

Hope that helps!

| cite | improve this answer | |
  • 3
    $\begingroup$ wo what loss function is suitable if input is floating between 0 and 1 ? $\endgroup$ – Jemshit Iskenderov Mar 21 '18 at 15:34
  • $\begingroup$ Binary Cross Entropy Loss $\endgroup$ – torayeff Jul 6 '19 at 19:24
  • $\begingroup$ Good point that binary cross entropy is asymmetric in the case when ground truth is not binary value(i.e. not 0 or 1, but 0.8 for example). But actually it works in practice blog.keras.io/building-autoencoders-in-keras.html but maybe less stable(?) for example changing optimizer to sgd lead to bad results for me. $\endgroup$ – mrgloom Jul 30 at 16:34
  • $\begingroup$ Also paper about auto encoders with BCE loss [On denoising autoencoders trained to minimise binary cross-entropy] arxiv.org/pdf/1708.08487.pdf $\endgroup$ – mrgloom Jul 31 at 10:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.