29
$\begingroup$

I am experimenting a bit autoencoders, and with tensorflow I created a model that tries to reconstruct the MNIST dataset.

My network is very simple: X, e1, e2, d1, Y, where e1 and e2 are encoding layers, d2 and Y are decoding layers (and Y is the reconstructed output).

X has 784 units, e1 has 100, e2 has 50, d1 has 100 again and Y 784 again.

I am using sigmoids as activation functions for layers e1, e2, d1 and Y. Inputs are in [0,1] and so should be the outputs.

Well, I tried using cross entropy as loss function, but the output was always a blob, and I noticed that the weights from X to e1 would always converge to an zero-valued matrix.

On the other hand, using mean squared errors as loss function, would produce a decent result, and I am now able to reconstruct the inputs.

Why is that so? I thought I could interpret the values as probabilities, and therefore use cross entropy, but obviously I am doing something wrong.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ You mentioned using sigmoid for the activation functions in the first layers. Those exhibit "saturation," as explained more fully in the CS231n Notes. Have you tried another activation function like ReLUs? $\endgroup$
    – Jerry
    Dec 1, 2017 at 14:12
  • $\begingroup$ I can't recall right now if I tried other, but thanks for sharing! $\endgroup$
    – AkiRoss
    Dec 3, 2017 at 21:14
  • $\begingroup$ I think the specific problem here pertains to using saturating non-linearities. But these related questions ask about using BCE loss for non-binary targets: stats.stackexchange.com/questions/370179/… and stats.stackexchange.com/questions/394582/… $\endgroup$
    – Sycorax ♦
    Jul 31, 2020 at 0:25

2 Answers 2

Reset to default
31
$\begingroup$

I think the best answer to this is that the cross-entropy loss function is just not well-suited to this particular task.

In taking this approach, you are essentially saying the true MNIST data is binary, and your pixel intensities represent the probability that each pixel is 'on.' But we know this is not actually the case. The incorrectness of this implicit assumption is then causing us issues.

We can also look at the cost function and see why it might be inappropriate. Let's say our target pixel value is 0.8. If we plot the MSE loss, and the cross-entropy loss $- [ (\text{target}) \log (\text{prediction}) + (1 - \text{target}) \log (1 - \text{prediction}) ]$ (normalising this so that it's minimum is at zero), we get:

cross-entropy vs. mse loss

We can see that the cross-entropy loss is asymmetric. Why would we want this? Is it really worse to predict 0.9 for this 0.8 pixel than it is to predict 0.7? I would say it's maybe better, if anything.

We could probably go into more detail and figure out why this leads to the specific blobs that you are seeing. I'd hazard a guess that it is because pixel intensities are above 0.5 on average in the region where you are seeing the blob. But in general this is a case of the implicit modelling assumptions you have made being inappropriate for the data.

Hope that helps!

Improve this answer
$\endgroup$
4
  • 3
    $\begingroup$ wo what loss function is suitable if input is floating between 0 and 1 ? $\endgroup$
    – Jemshit
    Mar 21, 2018 at 15:34
  • $\begingroup$ Binary Cross Entropy Loss $\endgroup$
    – torayeff
    Jul 6, 2019 at 19:24
  • $\begingroup$ Good point that binary cross entropy is asymmetric in the case when ground truth is not binary value(i.e. not 0 or 1, but 0.8 for example). But actually it works in practice blog.keras.io/building-autoencoders-in-keras.html but maybe less stable(?) for example changing optimizer to sgd lead to bad results for me. $\endgroup$
    – mrgloom
    Jul 30, 2020 at 16:34
  • $\begingroup$ Also paper about auto encoders with BCE loss [On denoising autoencoders trained to minimise binary cross-entropy] arxiv.org/pdf/1708.08487.pdf $\endgroup$
    – mrgloom
    Jul 31, 2020 at 10:29
3
$\begingroup$

I went through an autoencoder example listed at https://colab.research.google.com/github/ageron/handson-ml2/blob/master/17_autoencoders_and_gans.ipynb. The author used the binary cross-entropy loss function, and it seemed to work fine. I replaced it with the mse loss function, and the results were similar. Note that the author used the selu activation function. (Also, the notebooks help visualize the 'predicted' outputs as well)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.