49
$\begingroup$

I was reading the paper ImageNet Classification with Deep Convolutional Neural Networks and in section 3 were they explain the architecture of their Convolutional Neural Network they explain how they preferred using:

non-saturating nonlinearity $f(x) = max(0, x). $

because it was faster to train. In that paper they seem to refer to saturating nonlinearities as the more traditional functions used in CNNs, the sigmoid and the hyperbolic tangent functions (i.e. $f(x) = tanh(x)$ and $f(x) = \frac{1}{1 + e^{-x}} = (1 + e^{-x})^{-1}$ as saturating).

Why do they refer to these functions as "saturating" or "non-saturating"? In what sense are these function "saturating" or "non-saturating"? What do those terms mean in the context of convolutional neural networks? Are they used in other areas of machine learning (and statistics)?

$\endgroup$
2
  • $\begingroup$ I also found this quora answer very helpful. $\endgroup$ Jul 4, 2019 at 21:18
  • 1
    $\begingroup$ It should be noted that the main important difference is not the form of the function and neither really its squashing behaviour but that the non-saturating ones don't have vanishing gradients if the activations for some reason get out of control. And you need gradients to do gradient descent. The Rectified Linear unit (ReLU), what you refer to by the max() formula, has a decent gradient at all values (pun intended ;). $\endgroup$
    – BjornW
    Jan 24, 2021 at 14:03

3 Answers 3

49
$\begingroup$

Intuition

A saturating activation function squeezes the input.


Definitions

  • $f$ is non-saturating iff $ (|\lim_{z\to-\infty} f(z)| = +\infty) \vee (|\lim_{z\to+\infty} f(z)| = +\infty) $
  • $f$ is saturating iff $f$ is not non-saturating.

These definitions are not specific to convolutional neural networks.


Examples

The Rectified Linear Unit (ReLU) activation function, which is defined as $f(x)=max(0,x)$ is non-saturating because $\lim_{z\to+\infty} f(z) = +\infty$:

enter image description here

The sigmoid activation function, which is defined as $f(x) = \frac{1}{1 + e^{-x}}$ is saturating, because it squashes real numbers to range between $[0,1]$:

enter image description here

The tanh (hyperbolic tangent) activation function is saturating as it squashes real numbers to range between $[-1,1]$:

enter image description here

(figures are from CS231n, MIT License)

$\endgroup$
6
  • 1
    $\begingroup$ ah, nice makes sense! I know this wasn't my original question, but what is that property important in the context of ML and CNNs? $\endgroup$ Sep 28, 2015 at 15:43
  • $\begingroup$ For ANNs, to avoid having one unit with a large output that impacts too much the ANN's output layer. $\endgroup$ Sep 28, 2015 at 18:03
  • $\begingroup$ whats the difference between tan and sigmoid? both of them squash the numbers in a closed range! I dont get it, Can you elaborate this abit more? I'm kind of bad in mathematics . (by the way I'm coming from a CNN perspective ) $\endgroup$
    – Hossein
    Feb 17, 2016 at 11:09
  • $\begingroup$ @FranckDernoncourt Did you mean saturating for tanh activation function? I guess there is a typo? :) $\endgroup$ Mar 24, 2016 at 4:44
  • 3
    $\begingroup$ @tenCupMaximum: To saturate means to fill up to a point where no more can be added. In the context of a saturating function, it means that after a certain point, any further increase in the function's input will no longer cause a (meaningful) increase in its output, which has (very nearly) reached its maximum value. The function at that point is "all filled up", so to speak (or saturated). $\endgroup$ Oct 10, 2017 at 11:16
1
$\begingroup$

The most common activation functions are LOG and TanH. These functions have a compact range, meaning that they compress the neural response into a bounded subset of the real numbers. The LOG compresses inputs to outputs between 0 and 1, the TAN H between -1 and 1. These functions display limiting behavior at the boundaries.

At the border the gradient of the output with respect to the input ∂yj/∂xj is very small. So Gradient is small hence small steps to convergence hence longer time to converge.

$\endgroup$
1
$\begingroup$

In the neural network context, the phenomenon of saturation refers to the state in which a neuron predominantly outputs values close to the asymptotic ends of the bounded activation function.

So, saturation refers to behaviour of a neuron in a neural network after a given period of training/for a given range of input, and only neurons with bounded limits are susceptible to saturation (and by extension, such functions are sometimes referred to as 'saturating' even if in a particular instance they have not 'saturated').

Saturating functions include:

Type Examples
Limited as x approaches infinity and minus infinity Sigmoid, tanh
Limited only in one direction $\max(x,c)$

Non saturating functions include:

Type Examples
Unbounded functions identity, $\sinh$, $abs$
Periodic functions sin, cos

So in your example, a "non-saturating nonlinearity" means a "non-linear function with no limit as x approaches infinity".

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.