2
$\begingroup$

I was reading a paper that used neural networks to predict protein conformation, Improving prediction of secondary structure, local backbone angles, and solvent accessible surface area of proteins by iterative deep learning:

Linear activation function was used for the hidden layers of auto encoder training whereas sigmoid activation function was employed at the stage of back propagation.

Little unclear, but I interpret this to mean they did their sparse autoencoder parameter initialization training with a linear activation, then did the supervised training with a sigmoidal activation.

What's the rationale for this? I assumed that NN weights were only meaningful in the context of a particular activation function, and the whole point of the autoencoder-y initialization was to make the weights relevant for the supervised training...?

$\endgroup$
1
$\begingroup$

The excerpt:

Deep neural-network learning

Here, we employed the same deep learning neural network as we have employed for prediction of Cα-based θ and τ angles prediction by SPIDER16. Briefly, the deep artificial Neural Network (ANN) consists of three hidden layers, each with 150 nodes. Input data was normalized to the range of 0 to 1. Weights for each layer were initialized in a greedy layer-wise manner, using stacked sparse auto-encoders which map the layer’s inputs back to themselves32 and refined using standard backward propagation. The learning rate for auto encoder stage was 0.05 and the number of epochs in auto encoder stage was 10. The learning rates for backward propagation were 1, 0.5, 0.2, and 0.05, respectively, with 30 epochs at each learning rate. In this study, we used the deep neural network MATLAB toolbox, implemented by Palm33. Linear activation function was used for the hidden layers of auto encoder training whereas sigmoid activation function was employed at the stage of back propagation. All these hyper parameters were obtained by a few initial studies of a single fold (90% for training and 10% for test), randomly selected from the training TR4590 dataset.

So yeah, I believe what you said is right. Their stacked autoencoder approach uses linear activations, while sigmoid activation is used in the final model.

What you said about using the same activation for both the pre-training and refinement makes sense. The inputs to the second layer should be bounded by the sigmoid activation of the previous layer, which is not respected when using linear activations.

Consider, however, that the pre-training does not aim to retrieve optimal weights for the supervised refinement. Its main objective is to supply meaningful starting values (the autoencoder captures inter-observation variability, which should be partially explainable by the output of the supervised model).

So, that's why, even though the distribution of the weights in the pre-training deviates from the distribution of the weights in the refinement phase, it's still possible to get good results out of the procedure solely from the fact that the pre-training put the network on track of good weights.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.