Although it seems clear that ReLU and/or leaky ReLU have advantages over sigmoid or tanh activation functions in many situations, I find it very difficult to find out whether the latter are really "legacy". Is there a common situation in which using tanh or sigmoid activations is better than both ReLU and leaky ReLU?

To clarify, "better" may mean faster or more stable training, a better model precision, or any other desirable quality (please explain which one it is in your example). With a "common situation" I mean it should be a bit broader than one particular exotic example which breaks down as soon as the hyperparameters are chosen slightly differently.


A common use case is multi-class classification. Using the sigmoid activation in the final layer produces a quantity in $[0,1]$. When used element-wise, the output is a vector where each element is a probability. This is in contrast to the softmax case, where the entire vector is a probability distribution over the classes. A vector where each element is a probability can be helpful for tasks where the target has multiple, non-exclusive categories.

Another use case for tanh and sigmoid activations is in LSTM units, where the activations' constraints form a critical part of the so-called "constant error carousel".

  • $\begingroup$ Indeed, I think there is a use-case for sigmoid in the last layer. But as far as I understand, that is especially for non-mutually exclusive classification. For classification with mutually exclusive classes, softmax is the usual choice. $\endgroup$ Nov 20 '18 at 8:14
  • $\begingroup$ @MightyCurious Good point. See my edit. $\endgroup$
    – Sycorax
    Nov 20 '18 at 14:26

I've found that for simple regression problems with neural networks, tanh can be superior to ReLU. An example would be input = x, output = sin(x), over a limited domain such as [-pi,pi]. Any function approximated with ReLU activation functions is going to be piecewise linear. So, it takes a lot of piecewise linear functions to fit to a smooth function like sin. Meanwhile, tanh is very smooth, and it doesn't take as many tanh primitives to build something that closely resembles a sine wave.

Note that for classification problems, especially using multiply convolution layers, ReLU and its minor variants are hard to beat.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.