What is the process of selecting the best activation function? I already know which functions are better for which kind of problem, but I don't know why a particular function is better for a particular problem.

Also, I saw a list of few activation functions. I assumed that those were selected by great minds but I don't know the process which helps me to find if a function can work as activation function. Like there may be some constraints which should be satisfied by a function to be an activation function.


There isn't a general process of selecting an activation function. A lot of it is emperical, once they satisfy certain properties..

Although, we are interested in having the network compute interesting functions, so if you were to use for example a linear function as activation function (i.e. $f(x) = x$) then you're network wouldn't be able to model non-lineariites that might be present in your dataset. As a result, you would want non-linearities in your activation function, which are present in ReLU, sigmoid, tanh, etc.

The reason you see ReLU $f(x) = max(0, x)$ being used by default is because it enabled gradients to flow when the input to the ReLU function is positive, and does not have the saturation problems of sigmoid/tanh. Then some subequent papers they saw that: "Oh! units die on the left half" (where it's flat), and so they introduced modifications like PReLU which is $f(x) = max(-\alpha x, x)$ so now you also have gradients flowing when the input is negative (i.e. learning won't stop).

Now as to which one you should use for your network, you would have to run experiments!

  • $\begingroup$ Thank you for the answer.. I already knew this but as you said the functions should satisfy some properties in order to be used as an activation function. I need to know that properties. Can you tell me? $\endgroup$ Aug 28 '18 at 5:56
  • $\begingroup$ You would want it to be 1. differentiable over a reasonable domain, 2. introduce non-linearity (otherwise the network wouldn't compute something interesting), $\endgroup$
    – lonel
    Sep 3 '18 at 18:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.