Choice of neural net hidden activation function I have read elsewhere that one's choice of hidden layer activation function in a NN should be based on one's need, i.e. if you need values in the range -1 to 1 use tanh and use sigmoid for the range 0 to 1.
My question is how does one know what one's need is? Is it based on the range of the input layer, e.g. use the function that can encompass the input layer's full range of values, or somehow reflects the input layer's distribution (Gaussian function)? Or is the need problem/domain specific and one's experience/judgement is required to make this choice? Or is it simply "use that which gives the best cross-validated minimum training error?"
 A: The need mentioned in the first paragraph of the question relates to the output layer activation function, rather than the hidden layer activation function.  Having outputs that range from 0 to 1 is convenient as that means they can directly represent probabilities.  However, IIRC, a network with tanh output layer activation functions can be trivially transformed into a network with logistic output layer activation function, so it doesn't really matter much in practice.
IIRC the reason for using tanh rather than logistic activation function in the hidden units, which is that change made to a weight using backpropagation depends on both the output of the hidden layer neuron and on the derivative of the activation function, so using the logistic activation function you can have both go to zero at the same time, which can end up with the hidden layer unit becoming frozen.  
In short, use tanh for hidden layer activation functions, chose the output layer activation function to enforce desired constraints on the output (common choices: linear - no constraints, logistic - output lies between 0 and 1 and exponential - output strictly positive).
A: You may use $1.7159 \times \tanh(x \times (2/3))$ on hidden layers. This sigmoid has the property that it has max of its second derivatives at $-1$ and $+1$ values while its asymptotic limits are $[-1.5,+1.5]$. In that way you network will be more accurate on the points near the decision boundary. 
The general concept to choose sigmoid for your purpose is to choose the one according to the rule, your output values are in the the range of points, makes the second derivative of sigmoid function maximum.
A: LeCun discusses this in Efficient Backprop Section 4.4. The motivation is similar to the motivation for normalizing the input to zero mean (Section 4.3). The average outputs of the tanh activation function are more likely to be close to zero than the sigmoid, whose average output must be positive.
