Let's say I am writing an algorithm for building 2-layers stacked autoencoder and 2-layers neural network. Are they the same things or difference?

What I understand is that when I build a stacked autoencoder, I would build layer by layer. For neural network, I would initialize all the parameters in the netowork, and then for each data point, I pass it through the network and calculate the loss (e.g., euclean distance) and do backpropagation.


1 Answer 1


Stacked autoencoders and the multi-layer neural networks are different. In practice, you'll have the two networks share weights and possibly share memory buffers. So in your implementation the two networks become entwined.

Typically, autoencoders are trained in an unsupervised, greedy, layer-wise fashion. (No labels, begin training with just the first layer of the network and then add new layers as you go.) The weights can be learned using a variety of techniques ranging from "batch" gradient descent (please don't do that), to mini-batch stochastic gradient descent (SGD), to quasi-Newton methods like L-BFGS.

The idea is that the weights learned in a unsupervised manner to minimize reconstruction error for the representation learning task offer a good starting point to initialize a network for a supervised discriminative task such as classification or similarity. I.e., the network learns something about the underlying distribution by looking at the unlabeled data, allowing it to discriminate between labeled data. However, the weights still need to be "fine-tuned" for this new task. So add a logistic regression layer on the top of the network and then do supervised learning with a labeled dataset. The fine tuning step will do gradient descent and adjust the weights for all layers in the network simultaneously.

The advantages to this way of training neural nets are:

  1. Unsupervised training lets you show the network more data because it's much easier to get large unsupervised datasets than it is to get labeled ones.
  2. You can use the pre-trained network as a "jumping off point" for training new classifiers so you don't have to start from scratch each time.

For the paper, see Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.

  • $\begingroup$ I am confused about the second step where you train the encoding layer with a classification layer (e.g. softmax layer) for a classification task. Do you re-use the training dataset again for this part to learn the labels? I find this odd because the training set was just used to learn an approximation of itself . $\endgroup$
    – guy
    Commented Dec 14, 2017 at 23:04
  • 1
    $\begingroup$ It doesn't make much sense if you have all labeled data anyways. It's more for the situations where you have much more unlabeled data to pretrain and then use the labeled "training" data to fine tune. $\endgroup$
    – meow
    Commented Apr 19, 2018 at 12:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.