1
$\begingroup$

If I have this Neural network (fully connected) where the inputs are values from [0-1] (actual floats, not binary classification). And the output is a float value from [0-1] (actual float, not binary classification). Does anyone know what I should set each neuron's activation function?

Is there some conventions or rules I can follow to decide?

Based on the specific library I am using, I have these options https://github.com/wagenaartje/neataptic/wiki/Activation

Thanks

enter image description here

$\endgroup$
  • $\begingroup$ You can use whatever you like. It depends on your problem not the topology of network. But for output node it is common to use logsig when output is (0,1) $\endgroup$ – Cagdas Ozgenc Apr 21 '17 at 18:48
  • $\begingroup$ But I can control the activation function per neuron. I was thinking logistic for output neuron, but is there conventions for hidden layers? Also do you know anything about RELU? $\endgroup$ – omega Apr 21 '17 at 18:50
  • $\begingroup$ As far as I know RELU is used in deep network architectures (deep learning) in order to avoid some problems such as vanishing gradient. But I am not the best person to ask. In general in intermediary layers tansig is the popular choice. $\endgroup$ – Cagdas Ozgenc Apr 21 '17 at 18:51
  • $\begingroup$ So if I set up the hidden layers to be TANH and the output layer to be LOGISTIC, would that be an acceptable choice? $\endgroup$ – omega Apr 21 '17 at 18:53
  • $\begingroup$ The options I can pick from is here github.com/wagenaartje/neataptic/wiki/Activation $\endgroup$ – omega Apr 21 '17 at 18:53
3
$\begingroup$

Judging from recent research papers, the most popular one is the relu. However, I personally had occasionally better results with elu, leaky relu, softsign or even tanh. The first two don't seem to be supported by your framework, but are listed on the excellent wikipedia page on activation functions.

It only depends a little on the topology. Here are my personal and completely subjective rules of thumb:

  1. For deep nets (= more than two layers of weights), tanh and softsign are less appropriate due to the saturating and hence vanishing gradients on both sides.
  2. The unbounded ones (relu, leaky relu, softplus) are less appropriate for recurrent architectures, as their activations can grow pretty fast pretty big. You need a more sensitive initialisation here, and still learning can diverge anytime during optimisation unless you use tricks.
  3. For relu, the gradient can get strictly zero. This sometimes leads to "dead units" which are always off and cannot recover. The elu, leaky relu and softplus don't have that problem.
  4. Overall, one typically choses the same activation function for all nodes. This is not because of some theoretical insight, I suppose, but laziness of the users.
  5. The actual choice of transfer function is rarely responsible for a huge jump in performance.

My advice is to cross-validate them–try all of them. If you need to settle on three, I'd go for leaky relu, elu and softsign.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.