What rules define a valid neural network activation function, excluding biological plausibility? What set of principles do softmax, rectified linear units, hyperbolic tangent, sigmoid, etc. follow?
"Valid" in this case means that it maintains the universal approximation capability.
It must be a function of its inputs. It can be univariate or multivariate (for example, softmax). The more useful ones are often non-constant and continuous.
(Approximately) Monotonic functions have been found, empirically, to be better.
A few very successful activations are not monotonic however (Swish for example).
It does not even need to be one to one (see the softmax again, it's many to many, and the concatenated RELU, CRELU, which is one to many).
I think continuity is pretty much a must. The more heterodox-but-still-highly-useful ones like Maxout (element-wise maximum between several parallel layers) are continuous.
First note that a neural network usually has more than one activation functions (the activation function in hidden layers is often different from that used in the output layer).
Any function that is continuous can be used as an activation function, including linear function g(z)=z, which is often used in an output layer.
Since only activation function can introduce nonlinearity to a network, and nonlinearity is indispensable for a neural network to make untrivial predictions, activation functions in hidden layers are usually nonlinear, e.g. relu, which is a piecewise linear function that introduces the most simple nonlinearity.
