I have read several papers on Convolutional Neural Nets but I am yet to come across any that has used thresholds on tanh or sigmoid to decide whether the neuron will fire or not. Obviously this works for ReLU, but why is it not used for tanh or sigmoid ? Having a 0.0001 coming out of an activation unit can do less good and much harm to a deep neural net because I am loosing sparsity.
On the other hand, I have encountered deep neural nets not able to use ReLU at all (specially in the fully connected layers) because it leads to explosion of gradients and un trainable nets. So I have to fall back on tanh and sigmoid.
Any advice will be really helpful Regards