Can I use *any* activation function in my Neural network as long as it is non-linear? I am currently learning about Neural networks and am finding it hard to comprehend the reason why activation functions such as ReLU or sigmoid are favored.
In general, I understand the need for the activation functions to be nonlinear to produce nonlinear decision boundaries. However, I am unable to grasp the rationale behind the NN community in choosing certain nonlinear functions over others.
 In my understanding, I should be able to plug-in say a cubic polynomial and train my NN and expect results that perform more or less to what a sigmoid function shall perform. 
Can you point out where my understanding might be wrong and refer some literature I can read on this question.
 A: One reason for focus on sigmoid and ReLU networks is the Universal Approximation Theorems for neural networks. UATs describe under what conditions a neural network can approximate which functions. There are usually lots of conditions required to obtain the desired approximation result. In other words, not all functions can be approximated to a desired precision under all circumstances. 
The Cybenko UAT proved the universal approximation result for sigmoid activations, which is one reason that most research (until recently) focused on shallow, sigmoidal networks.
More recently, attention has turned to ReLU networks, for which there are several approximation theorems with their own requirements.
I don't have lots of details about UATs but as far as I know, there is not a UAT for non-monotonic functions, so that would seem to exclude certain cubic polynomials from being used as activation functions. (This does not stop certain non-monotonic functions from achieving nice results; the non-monotonic Mish activation function recently had some success in an image classification task.)
From a practical standpoint, cubics would seem to be a poor choice because they would tend to encourage exploding and vanishing gradients, especially in deep networks. ReLUs are populate because when the gradient is nonzero, it is fixed at 1, so gradients neither explode nor vanish as network depth increases. In general, a polynomial would seem to have gradients that increase or decrease very rapidly, so you would have to do some work to manage the magnitude of the gradient during training.
While the name, "universal approximation theorem" sounds very impressive, it's important to note that UATs do not provide results for how easy it is to train a certain network. The optimization task is hard; the limitations of data collection imply limitations on network quality; the number of units required for a specific network design to attain a certain level of quality may be impractically large; etc.
It's not hard to imagine non-linear activation functions which would be problematic for a neural network. One function is the signum function. The gradient is 0 except at 0 (where it is not defined), so it will not be possible to train a network using back-propagation because the weights will never update.

G. Cybenko. "Approximation by superpositions of a sigmoidal function." Mathematics of Control, Signals, and Systems (MCSS), 2(4):303–314, December 1989
Lu, Z., Pu, H., Wang, F., Hu, Z., & Wang, L. (2017). "The Expressive Power of Neural Networks: A View from the Width." Neural Information Processing Systems, 6231-6239.
Hanin, B. (2018). "Approximating Continuous Functions by ReLU Nets of Minimal Width." arXiv preprint arXiv:1710.11278.
A: I would say it is both a yes and a no. Ignoring impossibility by design, say like predicting negative output but with final ReLU activation, Yes, we could argue there might exist a configuration that might generate a "satisfactory" approximation. 
Now come the second problem, how do we reach there? Neural Network, must also be seen from optimization stand-point which begs the question given some initialized weights, would our model be able to converge to a satisfactory solution? I think if you have dealt with neural network quite sometime you might heard problems like vanishing or exploding gradient this is something that your activation function of choice should be able to at least handle. And hence a big No.
