I am training a simple autoencoder in Keras. The input is of length two, where each element can either be 0 or 1. This gives four distinct input possibilities: [0, 0], [0, 1], [1, 0], [1, 1]. Since there is no noise present, the input can only be these four options.
It does not really make sense to use an autoencoder here, but I am trying to build a basic understanding and make sure my general approach works for simple problems with known outcomes before proceeding to more complex problems that involve adding noise, etc.
I realize this is applied to images so may not directly translate, but I used this tutorial as a starting point: https://blog.keras.io/building-autoencoders-in-keras.html. Here is the structure I used (I made this up, not a ton of intuition behind this architecture choice but wanted to keep it simple and take into account the input/output data format).
input_dim = 2 encoded_dim = 4 input_ = Input(shape=(input_dim,)) encoded_ = Dense(encoded_dim, activation='relu')(input_) decoded = Dense(input_dim, activation='relu')(encoded_) output = Dense(input_dim, activation='sigmoid')(decoded) # because output should be between 0 and 1 autoencoder = Model(input_, output) autoencoder.compile(optimizer='adam',loss='binary_crossentropy') # because each output node in the autoencoder should be a 0 or 1 corresponding to reconstructed input autoencoder.fit(inputs, inputs, batch_size = 8, epochs=12, validation_split = 0.20)
However, I am finding the loss to be extremely high and that the predicted output does not correspond at all the input. However, I find this extremely surprising because the problem itself is extremely simple and does not require NNs to solve efficiently since there is no noise/variability in the data. In fact, setting most of the weights to 0 and a few to 1 solves the problem. Do you have any thoughts on why this problem might be difficult for a NN to solve or why my specific approach (architecture choice, loss or activation functions) would not work for this problem? I would appreciate any feedback/improvements/suggestions to help learn. Thank you!