This wikipedia article for Autoencoders states the following [07.30am UTC, 28th November 2017]:
An autoencoder is often trained using one of the many variants of backpropagation (such as conjugate gradient method, steepest descent, etc.). Though these are often reasonably effective, there are fundamental problems with the use of backpropagation to train networks with many hidden layers. Once errors are backpropagated to the first few layers, they become minuscule and insignificant. This means that the network will almost always learn to reconstruct the average of all the training data.
I'm aware that there used to be problems with training very deep networks, but I thought that the issue was overfitting occurring due to the inability to regularize properly - something that dropout and other more recent methods (e.g. batch norm, data augmentation...) have resolved.
I have heard of the problem of "Vanishing Gradients" in the context of Recurrent Neural Networks. Is this the same phenomenon? Have we actually overcome this problem somehow, or are small gradients no longer an issue because we have such powerful computational capacity with modern GPUs?