This wikipedia article for Autoencoders states the following [07.30am UTC, 28th November 2017]:

An autoencoder is often trained using one of the many variants of backpropagation (such as conjugate gradient method, steepest descent, etc.). Though these are often reasonably effective, there are fundamental problems with the use of backpropagation to train networks with many hidden layers. Once errors are backpropagated to the first few layers, they become minuscule and insignificant. This means that the network will almost always learn to reconstruct the average of all the training data.

I'm aware that there used to be problems with training very deep networks, but I thought that the issue was overfitting occurring due to the inability to regularize properly - something that dropout and other more recent methods (e.g. batch norm, data augmentation...) have resolved.

I have heard of the problem of "Vanishing Gradients" in the context of Recurrent Neural Networks. Is this the same phenomenon? Have we actually overcome this problem somehow, or are small gradients no longer an issue because we have such powerful computational capacity with modern GPUs?

  • 4
    $\begingroup$ Yes, it is the same phenomenon. Vanishing/exploding gradient is still a fundamental issue for Deep Neural Networks, especially when the deeper the network gets. Gradients either decay or blow up exponentially and the model will get slower to learn and tune the parameters of the earlier layers in the network. Solving the problem is still an active search in the literature because it is difficult to detect the problem and there is no one single way to overcome it. However, there are a lot of different approaches to deal with this problem. Using powerful computational engines is one of them. $\endgroup$ – ARAT Dec 3 '17 at 1:16
  • $\begingroup$ @ARAT I've copied your comment into an answer. If you would like to write up your comment as an answer, please let me know and I will delete my answer. $\endgroup$ – Sycorax Jul 4 '18 at 17:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.