# Do we still need to use tanh and sigmoid activation functions in neural networks, or can we always replace them by ReLU or leaky ReLU?

Although it seems clear that ReLU and/or leaky ReLU have advantages over sigmoid or tanh activation functions in many situations, I find it very difficult to find out whether the latter are really "legacy". Is there a common situation in which using tanh or sigmoid activations is better than both ReLU and leaky ReLU?

To clarify, "better" may mean faster or more stable training, a better model precision, or any other desirable quality (please explain which one it is in your example). With a "common situation" I mean it should be a bit broader than one particular exotic example which breaks down as soon as the hyperparameters are chosen slightly differently.

A common use case is multi-class classification. Using the sigmoid activation in the final layer produces a quantity in $$[0,1]$$. When used element-wise, the output is a vector where each element is a probability. This is in contrast to the softmax case, where the entire vector is a probability distribution over the classes. A vector where each element is a probability can be helpful for tasks where the target has multiple, non-exclusive categories.