1
$\begingroup$

I'm playing with the Neural Network toolbox in MATLAB. I've noted that each activation function (aka, transfer function) has 2 properties:

  • the output range, which, if I understand, is the codomain of the function, and
  • the active input range, which I really don't understand

For instance:

  • tansig (hyperbolic tangent sigmoid) has an output range of [-1,1] and an active input range of [-2,2].
  • logsig (log-sigmoid) has an output range of [0,1] and an active input range of [-4,4].
  • purelin (linear) has both an output range and an active input range of [-inf,+inf].
  • ...

I'm really confused...

So, can you tell me:

  • What is it the active input range of an activation function
  • How can I compute the active input range for a custom activation function?

Thank you so much for your time.

$\endgroup$

2 Answers 2

3
$\begingroup$

What seems to be meant by this definition is the range within which there is variation in the activation function (ie. where the activation is not saturated). For instance, for a tanh activation function, outside of $[-2,2]$, the activation function does not vary much, ie. its gradient is almost zero.

To compute this window, you can for instance compute the derivative of the activation function, choose a threshold $\epsilon$ under which you consider the derivative to be "small" (this is somewhat arbitrary, but values less than $10^{-2}$ of the derivative for an activation function taking values around $-1,1$ usually means relative "flatness".)

This matters for several reasons :

  • Since neural nets are usually trained using backpropagation, examples with activations falling in the saturated range (ie. outside the "active input range") of a neuron have no effect on the parameters of said neuron when computing the gradient (the gradient is essentially zero).

  • If say your features take extremely high values, and you initialize network weights at high values, a tanh unit for instance may be completely saturated for all examples when beginning training, and thus the network will not train at all. So you must take this saturation range into account when 1) scaling inputs and 2) initializing weights.

  • Generally, activation functions which do not saturate too much (RELu for instance) result in much faster & efficient training than saturating functions (sigmoid, tanh), precisely for the reasons above : consistently significant gradients and no saturation.

$\endgroup$
1
  • $\begingroup$ While still somewhat empirical (as you've also pointed out, "small" is arbitrary), looking at the gradient of the activation function makes things more clear. Also, now I understand why the active input range is used in the Nguyen-Widrow weight initialization function. Thanks $\endgroup$
    – seg.fault
    Commented Mar 3, 2015 at 14:16
0
$\begingroup$

If you look at a plot of an activation function e.g. https://en.wikipedia.org/wiki/File:Logistic-curve.svg you will see that any x-axis values outside the range -4 to 4 will have no appreciable difference in the resulting y-axis value - hence the active range is -4 to 4 because values within this range do have an appreciable difference.

$\endgroup$
1
  • $\begingroup$ So, if I understand, you say that the active input range of f(x) is [x0,x1] s.t. |f(x)-f(x1)|<e, for x>x1. and |f(x)-f(x0)|<e, for x<x0, where e>0. What should be the value of e? Also, for the hard-limit function, it seems that this definition doesn't apply as its active input range is [0,0]. $\endgroup$
    – seg.fault
    Commented Mar 3, 2015 at 12:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.