# How to select activation functions for neural networks?

What is the process of selecting the best activation function? I already know which functions are better for which kind of problem, but I don't know why a particular function is better for a particular problem.

Also, I saw a list of few activation functions. I assumed that those were selected by great minds but I don't know the process which helps me to find if a function can work as activation function. Like there may be some constraints which should be satisfied by a function to be an activation function.

Although, we are interested in having the network compute interesting functions, so if you were to use for example a linear function as activation function (i.e. $f(x) = x$) then you're network wouldn't be able to model non-lineariites that might be present in your dataset. As a result, you would want non-linearities in your activation function, which are present in ReLU, sigmoid, tanh, etc.
The reason you see ReLU $f(x) = max(0, x)$ being used by default is because it enabled gradients to flow when the input to the ReLU function is positive, and does not have the saturation problems of sigmoid/tanh. Then some subequent papers they saw that: "Oh! units die on the left half" (where it's flat), and so they introduced modifications like PReLU which is $f(x) = max(-\alpha x, x)$ so now you also have gradients flowing when the input is negative (i.e. learning won't stop).