In LeCun et. all "Deep Learning", Chapter 14, page 506, I found the following statement:
"A common strategy for training a deep autoencoder is to greedily pretrain the deep architecture by training a stack of shallow autoencoders, so we often encounter shallow autoencoders, even when the ultimate goal is to train a deep autoencoder."
I was just following the Keras tutorial on autoencoders, and they have a section on how to code up a deep autoencoder in Keras. I'm reproducing the code they give (using the MNIST dataset) below:
input_img = Input(shape=(784,)) encoded = Dense(128, activation='relu')(input_img) encoded = Dense(64, activation='relu')(encoded) encoded = Dense(32, activation='relu')(encoded) decoded = Dense(64, activation='relu')(encoded) decoded = Dense(128, activation='relu')(decoded) decoded = Dense(784, activation='sigmoid')(decoded) autoencoder = Model(input_img, decoded) autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy') autoencoder.fit(x_train, x_train, epochs=100, batch_size=256, shuffle=True, validation_data=(x_test, x_test))
My questions are:
- Does the code above represent stacked autoencoders or a deep autoencoder?
- If it is a deep autoencoder, how would you alter the above code to instead produce a stacked autoencoder? And vice versa?
- What is the advantage of one approach vs another?