I apologize in advance for the fact that I'm still coming up to speed on this. I'm trying to understand the pros and cons of using tanh (map -1 to 1) vs. sigmoid (map 0 to 1) for my neuron activation function. From my reading it sounded like a minor thing with marginal differences. In practice for my problems I find that the sigmoid is easier to train and strangely, the sigmoid appears to find general solution better. By this I mean that when the sigmoid version is done training it does well on the reference (untrained) data set, where the tanh version seems to be able to get the correct answers on training data while doing poorly on the reference. This is for the same network architecture.
One intuition I have is that with the sigmoid, it's easier for a neuron to almost fully turn off, thus providing no input to subsequent layers. The tanh has a harder time here since it needs to perfectly cancel its inputs, else it always gives a value to the next layer. Maybe this intuition is wrong though.
Long post. Bottom line, what's the trade, and should it make a big difference?