For my greedy layer-wise pre-training using sparse autoencoder, the first layer training seems to be okay since it can fairly reconstruct my test set.
However, because I use "sparse" autoencoder, the activation in the hidden layer is generally very low (0.01 on average) and a few hidden nodes activate at around 0.3 occasionally.
The problem occur when I have to do the second layer pre-training on top of those sparse hidden activation. I cannot whiten the data as I did directly to the raw input in the first layer.
What is a good way to deal with this problem?