I have read elsewhere that one's choice of hidden layer activation function in a NN should be based on one's need, i.e. if you need values in the range -1 to 1 use tanh and use sigmoid for the range 0 to 1.
My question is how does one know what one's need is? Is it based on the range of the input layer, e.g. use the function that can encompass the input layer's full range of values, or somehow reflects the input layer's distribution (Gaussian function)? Or is the need problem/domain specific and one's experience/judgement is required to make this choice? Or is it simply "use that which gives the best cross-validated minimum training error?"
1 + (1 / exp(-sum))
. Making the need very difficult to understand without trying both on each data set. The need as you describe it here is tied to the actual relation being learned, ie a binary data set will learn faster or not at all given different activations. $\endgroup$ – Adrian Seeley May 4 '14 at 16:47