In LeCun et. all "Deep Learning", Chapter 14, page 506, I found the following statement:
"A common strategy for training a deep autoencoder is to greedily pretrain the deep architecture by training a stack of shallow autoencoders, so we often encounter shallow autoencoders, even when the ultimate goal is to train a deep autoencoder."
I was just following the Keras tutorial on autoencoders, and they have a section on how to code up a deep autoencoder in Keras. I'm reproducing the code they give (using the MNIST dataset) below:
input_img = Input(shape=(784,))
encoded = Dense(128, activation='relu')(input_img)
encoded = Dense(64, activation='relu')(encoded)
encoded = Dense(32, activation='relu')(encoded)
decoded = Dense(64, activation='relu')(encoded)
decoded = Dense(128, activation='relu')(decoded)
decoded = Dense(784, activation='sigmoid')(decoded)
autoencoder = Model(input_img, decoded)
autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')
autoencoder.fit(x_train, x_train,
epochs=100,
batch_size=256,
shuffle=True,
validation_data=(x_test, x_test))
My questions are:
- Does the code above represent stacked autoencoders or a deep autoencoder?
- If it is a deep autoencoder, how would you alter the above code to instead produce a stacked autoencoder? And vice versa?
- What is the advantage of one approach vs another?