For my greedy layer-wise pre-training using sparse autoencoder, the first layer training seems to be okay since it can fairly reconstruct my test set.

However, because I use "sparse" autoencoder, the activation in the hidden layer is generally very low (0.01 on average) and a few hidden nodes activate at around 0.3 occasionally.

The problem occur when I have to do the second layer pre-training on top of those sparse hidden activation. I cannot whiten the data as I did directly to the raw input in the first layer.

What is a good way to deal with this problem?


This isn't specific to autoencoders, but a number of papers suggest procedures for initializing weights such that the outputs of each layer maintain a desired distribution. The motivation is similar to that of normalizing the inputs. Of course, the right procedure depends on the activation function.

Here are a few:

Glorot and Bengio (2010). Understanding the difficulty of training deep feedforward neural networks.

He et al. (2015). Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification.

LeCun et al. (1998). Efficient BackProp.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.