Do we still need to use tanh and sigmoid activation functions in neural networks, or can we always replace them by ReLU or leaky ReLU?

Question

Although it seems clear that ReLU and/or leaky ReLU have advantages over sigmoid or tanh activation functions in many situations, I find it very difficult to find out whether the latter are really "legacy". Is there a common situation in which using tanh or sigmoid activations is better than both ReLU and leaky ReLU?

To clarify, "better" may mean faster or more stable training, a better model precision, or any other desirable quality (please explain which one it is in your example). With a "common situation" I mean it should be a bit broader than one particular exotic example which breaks down as soon as the hyperparameters are chosen slightly differently.

Answer 1

A common use case is multi-class classification. Using the sigmoid activation in the final layer produces a quantity in $[0,1]$. When used element-wise, the output is a vector where each element is a probability. This is in contrast to the softmax case, where the entire vector is a probability distribution over the classes. A vector where each element is a probability can be helpful for tasks where the target has multiple, non-exclusive categories.

Softmax activations also play a crucial role in Transformer networks, which are a recent neural network architecture that has strong performance on sequence transduction tasks. See: "Attention Is All You Need" by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin. Replacing softmax with ReLU would create some problems, because then the output of the activation would not have a sum-to-1 constraint.

Another use case for tanh and sigmoid activations is in LSTM units, where the activations' constraints form a critical part of the so-called "constant error carousel".

Answer 2

I've found that for simple regression problems with neural networks, tanh can be superior to ReLU. An example would be input = x, output = sin(x), over a limited domain such as [-pi,pi]. Any function approximated with ReLU activation functions is going to be piecewise linear. So, it takes a lot of piecewise linear functions to fit to a smooth function like sin. Meanwhile, tanh is very smooth, and it doesn't take as many tanh primitives to build something that closely resembles a sine wave.

Note that for classification problems, especially using multiply convolution layers, ReLU and its minor variants are hard to beat.