3
$\begingroup$

What rules define a valid neural network activation function, excluding biological plausibility? What set of principles do softmax, rectified linear units, hyperbolic tangent, sigmoid, etc. follow?

"Valid" in this case means that it maintains the universal approximation capability.

$\endgroup$
3
  • $\begingroup$ Strictly speaking, you have to define your word 'valid'. Otherwise, any function can be an activation function. However, those you mentioned have in common that they are monotonically increasing (which isn't a must, as far I can see). $\endgroup$ – davidhigh Sep 18 '14 at 23:07
  • 1
    $\begingroup$ Thank you. I defined "valid." What makes you think that "monotonically increasing" is "not a must"? $\endgroup$ – user1559027 Sep 18 '14 at 23:21
  • $\begingroup$ If you are thinking about training it by backprop, maybe add an extra condition that the network be optimizable by backpropagation. In that case, i think continuous and piecewise differentiable may suffice. But not sure if what the math rigor version should be. $\endgroup$ – kawingkelvin Feb 26 '19 at 22:28
0
$\begingroup$

This wiki page 1 answers the question succintly. The really short answer is that the function only needs to be "nonconstant and bounded function"2

$\endgroup$
1
  • 1
    $\begingroup$ I think we are over the "bounded" part now. $\endgroup$ – Firebug Jan 13 '20 at 21:51
2
$\begingroup$

It must be a function of its inputs. It can be univariate or multivariate (for example, softmax). The more useful ones are often non-constant and continuous.

(Approximately) Monotonic functions have been found, empirically, to be better.

A few very successful activations are not monotonic however (Swish for example).

It does not even need to be one to one (see the softmax again, it's many to many, and the concatenated RELU, CRELU, which is one to many).

I think continuity is pretty much a must. The more heterodox-but-still-highly-useful ones like Maxout (element-wise maximum between several parallel layers) are continuous.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.