4
$\begingroup$

What rules define a valid neural network activation function, excluding biological plausibility? What set of principles do softmax, rectified linear units, hyperbolic tangent, sigmoid, etc. follow?

"Valid" in this case means that it maintains the universal approximation capability.

$\endgroup$
3
  • $\begingroup$ Strictly speaking, you have to define your word 'valid'. Otherwise, any function can be an activation function. However, those you mentioned have in common that they are monotonically increasing (which isn't a must, as far I can see). $\endgroup$
    – davidhigh
    Sep 18, 2014 at 23:07
  • 1
    $\begingroup$ Thank you. I defined "valid." What makes you think that "monotonically increasing" is "not a must"? $\endgroup$ Sep 18, 2014 at 23:21
  • $\begingroup$ If you are thinking about training it by backprop, maybe add an extra condition that the network be optimizable by backpropagation. In that case, i think continuous and piecewise differentiable may suffice. But not sure if what the math rigor version should be. $\endgroup$ Feb 26, 2019 at 22:28

3 Answers 3

0
$\begingroup$

This wiki page 1 answers the question succintly. The really short answer is that the function only needs to be "nonconstant and bounded function"2

$\endgroup$
1
  • 2
    $\begingroup$ I think we are over the "bounded" part now. $\endgroup$
    – Firebug
    Jan 13, 2020 at 21:51
2
$\begingroup$

It must be a function of its inputs. It can be univariate or multivariate (for example, softmax). The more useful ones are often non-constant and continuous.

(Approximately) Monotonic functions have been found, empirically, to be better.

A few very successful activations are not monotonic however (Swish for example).

It does not even need to be one to one (see the softmax again, it's many to many, and the concatenated RELU, CRELU, which is one to many).

I think continuity is pretty much a must. The more heterodox-but-still-highly-useful ones like Maxout (element-wise maximum between several parallel layers) are continuous.

$\endgroup$
0
0
$\begingroup$

First note that a neural network usually has more than one activation functions (the activation function in hidden layers is often different from that used in the output layer).

Any function that is continuous can be used as an activation function, including linear function g(z)=z, which is often used in an output layer.

Since only activation function can introduce nonlinearity to a network, and nonlinearity is indispensable for a neural network to make untrivial predictions, activation functions in hidden layers are usually nonlinear, e.g. relu, which is a piecewise linear function that introduces the most simple nonlinearity.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.