4
$\begingroup$

I am following this keras tutorial to create an autoencoder using the MNIST dataset. Here is the tutorial: https://blog.keras.io/building-autoencoders-in-keras.html.

However, I am confused with the choice of activation and loss for the simple one-layer autoencoder (which is the first example in the link). Is there a specific reason sigmoid activation was used for the decoder part as opposed to something such as relu? I am trying to understand whether this is a choice I can play around with, or if it should indeed be sigmoid, and if so why? Similarily, I understand the loss is taken by comparing each of the original and predicted digits on a pixel-by-pixel level, but I am unsure why the loss is binary crossentropy as opposed to something like mean squared error.

I would love clarification on this to help me move forward! Thank you!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You are correct that MSE is often used as a loss in these situations. However, the Keras tutorial (and actually many guides that work with MNIST datasets) normalizes all image inputs to the range [0, 1]. This occurs on the following two lines:

x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.

Note: as grayscale images, each pixel takes on an intensity between 0 and 255 inclusive.

Therefore, BCE loss is an appropriate function to use in this case. Similarly, a sigmoid activation, which squishes the inputs to values between 0 and 1, is also appropriate. You'll notice that under these conditions, when the decoded image is "close" to the encoded image, BCE loss will be small. I found more information about this here.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I wrote about it here, but it was ages ago so I cannot find it now; BCE's properties as a function means it's not the best choice for image data, even in greyscale. Unlike MSE, it is asymmetrically biased against overconfidence, so it systematically underestimates the values, needlessly dimming the output intensities. And, as this question shows, causes unnecessary confusion on top. $\endgroup$
    – jkm
    Commented Jan 4, 2020 at 21:41
  • $\begingroup$ Hmm. I think you may be correct in general, but for this particular use case (an autoencoder), it's been empirically and mathematically shown that training on the BCE and MSE objective both yield the same optimal reconstruction function: arxiv.org/pdf/1708.08487.pdf — but that's just a minor detail. $\endgroup$
    – chang_trenton
    Commented Jan 5, 2020 at 1:46
  • $\begingroup$ I cannot load the pdf for some reason, but I'm not surprised - the minima of both losses are the same if your goal is to autoencode a 1:1 match of intensities. It's just not always an optimal loss if your goal is to have a nice-looking image; e.g. MNIST would probably look best with most pixels being either 1 or 0 (in/not in the set of pixels for the character, basically learning a topology). $\endgroup$
    – jkm
    Commented Jan 5, 2020 at 11:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.