1
$\begingroup$

In order to extract some useful features from raw data (purely unsupervised tasks), we could use autoencoders. To me, there seem to be two approaches of training.

  1. We could do the training layer by layer and get a stacked auto encoder, say the raw data has 100 variables, the first autoencoder is developed using 50 hidden units. The resulting 50 activations are used as input for the second auto encoder, say using 10 hidden units. So now we have a stacked autoencoder and the 10 activations can be regarded as 10 new features for further use.

  2. Meanwhile, we could simply use a deep auto encoder architecture, and the architecture is like 100-50-10-50-100, where 50-10-50 are the hidden units in three hidden layers. Again, we could extract the 10 activations in the middle as new features.

I am wondering what's the essential difference between these two approaches. Which one is better in learning features? Thanks in advance for any help.

$\endgroup$
2
$\begingroup$

Unsupervised, layer-wise pretraining was one of the early innovations that made it possible to use deep networks in practice. Since then, other tricks have been discovered that made layerwise pre-training unnecessary in many cases. Rectified linear units (ReLUs) are one example.

Glorot et al. (2011). Deep Sparse Rectifier Neural Networks.

Using deep autoencoders with ReLUs, they found that unsupervised pretraining was unnecessary and, in some cases, performance was better without it. But, they did find that unsupervised pretraining can help in a semi-supervised setting, when unlabeled data is available.

Optimization methods are another class of tricks. For example, Hessian-free (HF) optimization uses second order information to compute the update directions. The following paper found that HF optimization made unsupervised pre-training unnecessary for training deep autoencoders.

Martens (2010). Deep learning via Hessian-free optimization.

My impression is that unsupervised, layer-wise pretraining has generally fallen out of favor, except for specific circumstances (e.g. the semisupervised case). For related discussion, see here.

$\endgroup$
0
$\begingroup$

the first one called deep-belief network based on the restricted boltzmann machine, where you can train each layer independently from the other and then using the output of this layer to train another layer. while the new method autoencoder allow you to build the same structure from the first time, here you have to know that the weights are just transposed between encode and decode. there is no one is better than other. DBN was old approach before AEN.

Try both of them and compare the results

$\endgroup$
  • 2
    $\begingroup$ Some of these points are incorrect. DBNs/RBMs are one example, but stacked autoencoders can also use standard sigmoidal or relu units. Setting the decoder weights equal to the transpose of the encoder weights is called 'tied weights', but a completely separate set of weights can also be used. $\endgroup$ – user20160 Jul 7 '16 at 12:33
  • $\begingroup$ where is the incorrect point. I said the initial idea behind AEN started by DBN. we didn't talk about sigm or Relu. again the transpose was also one idea. $\endgroup$ – Feras Jul 7 '16 at 14:25
  • $\begingroup$ Could just be a wording issue, but I interpret your post as suggesting that DBNs and stacked autoencoders are conceptually equivalent. Stacked AEs using sigmoid units and relus are a counterexample. It also sounded like you were saying that non-stacked AEs imply tied weights, in contrast to stacked AEs. But, both types of weights can be used in both stacked and non-stacked AEs. If that's not what you meant, that's ok, and an edit might help clarify things. No harshness intended, just looking for clarity. btw, let me give you a warmer welcome; it's nice to see new contributions. $\endgroup$ – user20160 Jul 7 '16 at 15:09
  • $\begingroup$ @Feras, thanks a lot for your reply. Personally I share the same feeling as user20160. As far as I know, tied weighted can be used in deep auto encoder but not necessarily. Also, it seems that we could not treat DBN and stacked autoencoder as equivalent. Thanks again for the reply. $\endgroup$ – rajafan Jul 8 '16 at 0:52
  • $\begingroup$ Hi @user20160, thanks for the comments. Do you mind share a slightly more detailed answer to this question? Thanks. $\endgroup$ – rajafan Jul 8 '16 at 0:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.