Deep neural-network learning
Here, we employed the same deep learning neural network as we have employed for prediction of Cα-based θ and τ angles prediction by SPIDER16. Briefly, the deep artificial Neural Network (ANN) consists of three hidden layers, each with 150 nodes. Input data was normalized to the range of 0 to 1. Weights for each layer were initialized in a greedy layer-wise manner, using stacked sparse auto-encoders which map the layer’s inputs back to themselves32 and refined using standard backward propagation. The learning rate for auto encoder stage was 0.05 and the number of epochs in auto encoder stage was 10. The learning rates for backward propagation were 1, 0.5, 0.2, and 0.05, respectively, with 30 epochs at each learning rate. In this study, we used the deep neural network MATLAB toolbox, implemented by Palm33. Linear activation function was used for the hidden layers of auto encoder training whereas sigmoid activation function was employed at the stage of back propagation. All these hyper parameters were obtained by a few initial studies of a single fold (90% for training and 10% for test), randomly selected from the training TR4590 dataset.
So yeah, I believe what you said is right. Their stacked autoencoder approach uses linear activations, while sigmoid activation is used in the final model.
What you said about using the same activation for both the pre-training and refinement makes sense. The inputs to the second layer should be bounded by the sigmoid activation of the previous layer, which is not respected when using linear activations.
Consider, however, that the pre-training does not aim to retrieve optimal weights for the supervised refinement. Its main objective is to supply meaningful starting values (the autoencoder captures inter-observation variability, which should be partially explainable by the output of the supervised model).
So, that's why, even though the distribution of the weights in the pre-training deviates from the distribution of the weights in the refinement phase, it's still possible to get good results out of the procedure solely from the fact that the pre-training put the network on track of good weights.