2
$\begingroup$

This question already has an answer here:

I'm toying around with autoencoders and tried the tutorial from the Keras blog (first section "Let's build the simplest possible autoencoder" only).

For the encoding layer they use ReLus, while sigmoids are used for the decoding layer. I wondered why that is and changed

encoded = Dense(encoding_dim, activation='relu')(input_img)

to

encoded = Dense(encoding_dim, activation='sigmoid')(input_img)

Instead of a val_loss of < 0.11 the model now reaches only ~0.26 after 50 epochs and the reconstruction looks way off too (first row are original digits, second row reconstructions)

I'm trying to understand why that is but can't find any information in the post. Does anyone know why changing the activation layer from relu to sigmoid has such a negative effect in this case?

$\endgroup$

marked as duplicate by Reinstate Monica, kjetil b halvorsen, Michael Chernick, Jakub Bartczuk, John Jun 19 '18 at 4:06

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

-3
$\begingroup$

This is pure intuition - nothing I can easily prove - but a ReLU has access to a far, far larger space of potential encodings.

With a Sigmoid, your encodings are constrained to a 0-1 range. While technically there is an infinite amount of numbers in that space, you run into issues with floating point arithmetic.

This means you have less possible encodings and it is easier to confuse two distinct encodings since they are squished together on the scale.

A ReLU can use any positive real number for its encodings, which makes it easier to space the encodings apart from each other.

This might have been somewhat mitigated in a deeper model, but if memory serves that tutorial uses just a single hidden layer, so there's not much room for error.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.