# $\tanh$ activation function and sparsity constraint

According to LeCun's paper "effient backprop"  the $\tanh$ activation function should be preferred over the logistic activation function for the hidden units in neural networks.

For the $\tanh$ units an output of $a_i = -1$ is considered as inactive. But is correct especially for the sparsity constraints (e.g. sparse autoencoders)? An inactive unit should act like not present. But an unit with $a_i = -1$ contributes to the $\text{logit}$ $\sum_i (w_ij * a_i)$ extremely. In contrast a unit with output $a_i = 0$ would not contribute to the sum. But for output $a_i = 0$ the $\tanh$ unit is absolutely not in the saturated regime.

What output is considered as inactive if there is an sparsity constraint to force most of the units to be inactive?

Is there a paper about the "sparsity of activation" for $\tanh$ units?

 Y. LeCun, L. Bottou, G. Orr and K. Muller: Efficient BackProp, in Orr, G. and Muller K. (Eds), Neural Networks: Tricks of the trade, Springer, 1998

It does not really matter. Let's say you wanted to fix this problem by using $f(x) = \tanh(x) + 1$ as a transfer, which makes its co-domain $(0, 2)$ but does not change the gradient characteristics which make it superior to the logistic.
Consider a very small network with a single input, output, weight and bias: $y = w f(x) + b$. This network is equivalent to the network $y = w \tanh(x) + b + 1$. This shows that you could also use a different bias $b' = b + 1$ letting you arrive at $y = w \tanh(x) + b'$.