# Do we still need to use tanh and sigmoid activation functions in neural networks, or can we always replace them by ReLU or leaky ReLU?

Although it seems clear that ReLU and/or leaky ReLU have advantages over sigmoid or tanh activation functions in many situations, I find it very difficult to find out whether the latter are really "legacy". Is there a common situation in which using tanh or sigmoid activations is better than both ReLU and leaky ReLU?

To clarify, "better" may mean faster or more stable training, a better model precision, or any other desirable quality (please explain which one it is in your example). With a "common situation" I mean it should be a bit broader than one particular exotic example which breaks down as soon as the hyperparameters are chosen slightly differently.

A common use case is multi-class classification. Using the sigmoid activation in the final layer produces a quantity in $$[0,1]$$. When used element-wise, the output is a vector where each element is a probability. This is in contrast to the softmax case, where the entire vector is a probability distribution over the classes. A vector where each element is a probability can be helpful for tasks where the target has multiple, non-exclusive categories.

Another use case for tanh and sigmoid activations is in LSTM units, where the activations' constraints form a critical part of the so-called "constant error carousel".

• Indeed, I think there is a use-case for sigmoid in the last layer. But as far as I understand, that is especially for non-mutually exclusive classification. For classification with mutually exclusive classes, softmax is the usual choice. Nov 20, 2018 at 8:14
• @MightyCurious Good point. See my edit.
– Sycorax
Nov 20, 2018 at 14:26

I've found that for simple regression problems with neural networks, tanh can be superior to ReLU. An example would be input = x, output = sin(x), over a limited domain such as [-pi,pi]. Any function approximated with ReLU activation functions is going to be piecewise linear. So, it takes a lot of piecewise linear functions to fit to a smooth function like sin. Meanwhile, tanh is very smooth, and it doesn't take as many tanh primitives to build something that closely resembles a sine wave.

Note that for classification problems, especially using multiply convolution layers, ReLU and its minor variants are hard to beat.