What does the term saturating nonlinearities mean?

Question

I was reading the paper ImageNet Classification with Deep Convolutional Neural Networks and in section 3 were they explain the architecture of their Convolutional Neural Network they explain how they preferred using:

non-saturating nonlinearity $f(x) = max(0, x). $

because it was faster to train. In that paper they seem to refer to saturating nonlinearities as the more traditional functions used in CNNs, the sigmoid and the hyperbolic tangent functions (i.e. $f(x) = tanh(x)$ and $f(x) = \frac{1}{1 + e^{-x}} = (1 + e^{-x})^{-1}$ as saturating).

Why do they refer to these functions as "saturating" or "non-saturating"? In what sense are these function "saturating" or "non-saturating"? What do those terms mean in the context of convolutional neural networks? Are they used in other areas of machine learning (and statistics)?

Answer 1

Intuition

A saturating activation function squeezes the input.

---

Definitions

• $f$ is non-saturating iff $ (|\lim_{z\to-\infty} f(z)| = +\infty) \vee (|\lim_{z\to+\infty} f(z)| = +\infty) $
• $f$ is saturating iff $f$ is not non-saturating.

These definitions are not specific to convolutional neural networks.

---

Examples

The Rectified Linear Unit (ReLU) activation function, which is defined as $f(x)=max(0,x)$ is non-saturating because $\lim_{z\to+\infty} f(z) = +\infty$:

The sigmoid activation function, which is defined as $f(x) = \frac{1}{1 + e^{-x}}$ is saturating, because it squashes real numbers to range between $[0,1]$:

The tanh (hyperbolic tangent) activation function is saturating as it squashes real numbers to range between $[-1,1]$:

(figures are from CS231n, MIT License)

Answer 2

In the neural network context, the phenomenon of saturation refers to the state in which a neuron predominantly outputs values close to the asymptotic ends of the bounded activation function.

• Measuring Saturation in Neural Networks (2015)

So, saturation refers to behaviour of a neuron in a neural network after a given period of training/for a given range of input, and only neurons with bounded limits are susceptible to saturation (and by extension, such functions are sometimes referred to as 'saturating' even if in a particular instance they have not 'saturated').

Saturating functions include:

Type	Examples
Limited as x approaches infinity and minus infinity	Sigmoid, tanh
Limited only in one direction	$\max(x,c)$

Non saturating functions include:

Type	Examples
Unbounded functions	identity, $\sinh$, $abs$
Periodic functions	sin, cos

So in your example, a "non-saturating nonlinearity" means a "non-linear function with no limit as x approaches infinity".

Answer 3

The most common activation functions are LOG and TanH. These functions have a compact range, meaning that they compress the neural response into a bounded subset of the real numbers. The LOG compresses inputs to outputs between 0 and 1, the TAN H between -1 and 1. These functions display limiting behavior at the boundaries.

At the border the gradient of the output with respect to the input ∂yj/∂xj is very small. So Gradient is small hence small steps to convergence hence longer time to converge.