1
$\begingroup$

I am experimenting with Neural networks for regression tasks. I know some theory, and how to choose the activation function for the output layer. What is not clear to me is: how to choose activation for hidden layers?

$\endgroup$

migrated from stackoverflow.com Dec 27 '18 at 14:30

This question came from our site for professional and enthusiast programmers.

3
$\begingroup$

With respect to choosing hidden layer activations, I don't think that there's anything about a regression task which is different from other neural network tasks: you should use nonlinear activations so that the model is nonlinear (otherwise, you're just doing a very slow, expensive linear regression), and you should use activations that are easy to train (ReLU or similar).

Recent research has found that ReLU and similar activations (ELU, Leaky ReLU, etc.) work very well because they allow researchers to build deep networks which do not suffer from vanishing or exploding gradient for positive inputs. See:

On the left, ReLU has derivative 0 and this can lead to the "dead ReLU" phenomenon. So I prefer using ELU or LeakyReLU units, which can be more robust to that problem.

$\endgroup$
0
$\begingroup$

Hyperbolic tangent and sigmoid transfer functions are usually implemented in the hiddens layers of the various neural network fitting tools in Matlab,R,ecc.. Their use is similar and you can try them very simply.They are both differentiable and continous.ReLU and similar are usually implemented in the convolutional neural networks but I don't think it is the first choice for your problem.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.