What are good activation functions for this neural net architecture? If I have this Neural network (fully connected) where the inputs are values from [0-1] (actual floats, not binary classification). And the output is a float value from [0-1] (actual float, not binary classification). Does anyone know what I should set each neuron's activation function?
Is there some conventions or rules I can follow to decide?
Based on the specific library I am using, I have these options https://github.com/wagenaartje/neataptic/wiki/Activation
Thanks

 A: Judging from recent research papers, the most popular one is the relu. However, I personally had occasionally better results with elu, leaky relu, softsign or even tanh. The first two don't seem to be supported by your framework, but are listed on the excellent wikipedia page on activation functions.
It only depends a little on the topology. Here are my personal and completely subjective rules of thumb:


*

*For deep nets (= more than two layers of weights), tanh and softsign are less appropriate due to the saturating and hence vanishing gradients on both sides.

*The unbounded ones (relu, leaky relu, softplus) are less appropriate for recurrent architectures, as their activations can grow pretty fast pretty big. You need a more sensitive initialisation here, and still learning can diverge anytime during optimisation unless you use tricks.

*For relu, the gradient can get strictly zero. This sometimes leads to "dead units" which are always off and cannot recover. The elu, leaky relu and softplus don't have that problem.

*Overall, one typically choses the same activation function for all nodes. This is not because of some theoretical insight, I suppose, but laziness of the users.

*The actual choice of transfer function is rarely responsible for a huge jump in performance.


My advice is to cross-validate them–try all of them. If you need to settle on three, I'd go for leaky relu, elu and softsign.
