$\tanh$ activation function and sparsity constraint

Question

According to LeCun's paper "effient backprop" [1] the $\tanh$ activation function should be preferred over the logistic activation function for the hidden units in neural networks.

For the $\tanh$ units an output of $a_i = -1$ is considered as inactive. But is correct especially for the sparsity constraints (e.g. sparse autoencoders)? An inactive unit should act like not present. But an unit with $a_i = -1$ contributes to the $\text{logit}$ $\sum_i (w_ij * a_i)$ extremely. In contrast a unit with output $a_i = 0$ would not contribute to the sum. But for output $a_i = 0$ the $\tanh$ unit is absolutely not in the saturated regime.

What output is considered as inactive if there is an sparsity constraint to force most of the units to be inactive?

Is there a paper about the "sparsity of activation" for $\tanh$ units?

[1] Y. LeCun, L. Bottou, G. Orr and K. Muller: Efficient BackProp, in Orr, G. and Muller K. (Eds), Neural Networks: Tricks of the trade, Springer, 1998

Answer 1

It does not really matter. Let's say you wanted to fix this problem by using $f(x) = \tanh(x) + 1$ as a transfer, which makes its co-domain $(0, 2)$ but does not change the gradient characteristics which make it superior to the logistic.

Consider a very small network with a single input, output, weight and bias: $y = w f(x) + b$. This network is equivalent to the network $y = w \tanh(x) + b + 1$. This shows that you could also use a different bias $b' = b + 1$ letting you arrive at $y = w \tanh(x) + b'$.

Long story short, the network will have the same output and the same gradients. The only difference would be if you regularised the bias with weight decay or such--but it is not a good idea to impose this kind of regularisation on the bias anyway.

What is much more interesting with respect to sparsity, is that the gradients for sparse units are near zero--this makes the output of the net not vary with the unit, albeit only linearly. This is also shown by the manifold tangent classifier network, where the units tend to become either 0 or 1--they try to saturate in order to make the network invariant to local distortions.