1
$\begingroup$

I am training a simple autoencoder in Keras. The input is of length two, where each element can either be 0 or 1. This gives four distinct input possibilities: [0, 0], [0, 1], [1, 0], [1, 1]. Since there is no noise present, the input can only be these four options.

It does not really make sense to use an autoencoder here, but I am trying to build a basic understanding and make sure my general approach works for simple problems with known outcomes before proceeding to more complex problems that involve adding noise, etc.

I realize this is applied to images so may not directly translate, but I used this tutorial as a starting point: https://blog.keras.io/building-autoencoders-in-keras.html. Here is the structure I used (I made this up, not a ton of intuition behind this architecture choice but wanted to keep it simple and take into account the input/output data format).

input_dim = 2
encoded_dim = 4
input_ = Input(shape=(input_dim,))
encoded_ = Dense(encoded_dim, activation='relu')(input_)
decoded = Dense(input_dim, activation='relu')(encoded_) 
output = Dense(input_dim, activation='sigmoid')(decoded) # because output should be between 0 and 1
autoencoder = Model(input_, output)

autoencoder.compile(optimizer='adam',loss='binary_crossentropy') # because each output node in the autoencoder should be a 0 or 1 corresponding to reconstructed input
autoencoder.fit(inputs, inputs, batch_size = 8, epochs=12, validation_split = 0.20)

However, I am finding the loss to be extremely high and that the predicted output does not correspond at all the input. However, I find this extremely surprising because the problem itself is extremely simple and does not require NNs to solve efficiently since there is no noise/variability in the data. In fact, setting most of the weights to 0 and a few to 1 solves the problem. Do you have any thoughts on why this problem might be difficult for a NN to solve or why my specific approach (architecture choice, loss or activation functions) would not work for this problem? I would appreciate any feedback/improvements/suggestions to help learn. Thank you!

$\endgroup$
  • 1
    $\begingroup$ encoded_dim is never used in your code... The network you use is hardly an autoencoder as it is missing any "bottleneck" part that would require "encoding" the information... This network could simply pass the data through (if it weren't for the sigmoid output, which can output 0 and 1 only if its inputs approach infinity). $\endgroup$ – Jan Kukacka Oct 22 '19 at 8:59
  • $\begingroup$ Apologies, I made fixed encoded_dim (my original code uses encoded_dim but I made a mistake when copying the code over). Is your point that encoded_dim > input_dim so it's not actually encoding anything? I probably have to make input_dim larger than 2 for this case, but is your suggestion to make encoded_dim < input_dim. $\endgroup$ – Jane Sully Oct 22 '19 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.