3
$\begingroup$

i am currently programming a convolutional autoencoder in python/tensorflow to reproduce the MNIST handwritten digits. Mostly i get pretty good results: The Autoencoder behaving as expected But in around 40% of the time when i start my program, the autoencoder(or more precise the decorder) is learning the average image of all the test data, and to not use the code in any way: Failed (The output of the Autoencoder on the right is the same for every input, and the code/the sliders have no impact at all)

  1. Why/How does this happen? I can imagine that there might be bad local minima or the network/training paramenters are choosen badly but it doesent make sense to me, that the learning process converges on a solution that doesent "use the code" at all.
  2. How can i fix it?

My Encoder consists of one convolution, then i flatten the result and after that a final fully connected Layer. The Decoder obviously doing the inverse. Here is the main part of my code:

import tensorflow as tf
import numpy as np
import Window
from PyQt5.QtWidgets import *
import sys


class Encoder(tf.keras.layers.Layer):
    def __init__(self, code_dim):
        super(Encoder, self).__init__()
        self.conv_layer = tf.keras.layers.Conv2D(
            filters=16, kernel_size=(3, 3), activation='relu', padding='same'
        )
        self.output_layer = tf.keras.layers.Dense(
            units=code_dim,
            activation=tf.nn.sigmoid
        )

    def call(self, input_features, **kwargs):
        convoluted = self.conv_layer(input_features)
        flattened = tf.reshape(convoluted, [convoluted.shape[0], convoluted.shape[1]*convoluted.shape[2]*convoluted.shape[3]])
        code = self.output_layer(flattened)
        return code


class Decoder(tf.keras.layers.Layer):
    def __init__(self):
        super(Decoder, self).__init__()
        self.input_layer = tf.keras.layers.Dense(
            units=12544,
            activation=tf.nn.sigmoid
        )
        self.deconv_layer = tf.keras.layers.Conv2DTranspose(
            1, (1, 1), strides=(1, 1), input_shape=(28, 28, 16)
        )

    def call(self, code, **kwargs):
        inp = self.input_layer(code)
        deflattened = tf.reshape(inp, [inp.shape[0], 28, 28, 16])
        out = self.deconv_layer(deflattened)
        return out


class Autoencoder(tf.keras.Model):
    def __init__(self, code_dim):
        super(Autoencoder, self).__init__()
        self.encoder = Encoder(code_dim=code_dim)
        self.decoder = Decoder()

    def call(self, input_features, **kwargs):
        code = self.encoder(input_features)
        reconstructedd = self.decoder(code)
        return reconstructedd


def loss(model, org):
    reconstruction_error = tf.reduce_mean(tf.square(tf.subtract(model(org), org)))
    return reconstruction_error


def train(lss, model, optt, origin):
    with tf.GradientTape() as tape:
        gradients = tape.gradient(lss(model, origin), model.trainable_variables)
        gradient_variables = zip(gradients, model.trainable_variables)
        optt.apply_gradients(gradient_variables)


gpus = tf.config.experimental.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(gpus[0], True)
epochs = 20
batch_size = 128
learning_rate = 0.01
codesize = 20

autoencoder = Autoencoder(code_dim=codesize)
opt = tf.optimizers.Adam(learning_rate=learning_rate)

(training_features, _), (test_features, _) = tf.keras.datasets.mnist.load_data()
training_features = training_features / np.max(training_features)
training_features = training_features.reshape(training_features.shape[0], training_features.shape[1], training_features.shape[2], 1)
training_features = training_features.astype('float32')
training_dataset = tf.data.Dataset.from_tensor_slices(training_features)
training_dataset = training_dataset.batch(batch_size)
training_dataset = training_dataset.shuffle(training_features.shape[0])
training_dataset = training_dataset.prefetch(batch_size * 4)

for epoch in range(epochs):
    print("Epoch", epoch)
    lastloss = 0
    for step, batch_features in enumerate(training_dataset):
        train(loss, autoencoder, opt, batch_features)
        loss_values = loss(autoencoder, batch_features)
        lastloss = loss_values
    print("  Endloss:", lastloss.numpy())

test_features = test_features / np.max(test_features)
test_features = test_features.reshape(test_features.shape[0], test_features.shape[1], test_features.shape[2], 1)
test_features = test_features.astype('float32')

# Window
app = QApplication(sys.argv)
window = Window.Window(autoencoder, test_features, codesize)
window.show()
app.exec_()
$\endgroup$

2 Answers 2

2
$\begingroup$

Good to see you figured out you initialized the weights to 0. But this shouldn't be a direct cause for the above effect by itself. The reason is that you have a posterior collapse, this means your decoder ignores whatever the encoder is providing and generates the same outcome.

A few ways to fix it is to change your bottle-neck or consider alternative architectures such as VAEs (that still are susceptible but should be less)

$\endgroup$
0
$\begingroup$

I found the cause: All weights were initalized with 0.

$\endgroup$
1
  • $\begingroup$ All of which weights? Why did initializing weights to 0 sometimes work (60% of the time, according to your post) and not work other times (40% of the time). In your post, you ask 2 questions. How would you answer each of them? $\endgroup$
    – Sycorax
    Feb 4, 2020 at 0:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.