How does rectilinear activation function solve the vanishing gradient problem in neural networks?

Question

I found rectified linear unit (ReLU) praised at several places as a solution to the vanishing gradient problem for neural networks. That is, one uses max(0,x) as activation function. When the activation is positive, it is obvious that this is better than, say, the sigmoid activation function, since its derivation is always 1 instead of an arbitrarily small value for large x. On the other hand, the derivation is exactly 0 when x is smaller than 0. In the worst case, when a unit is never activated, the weights for this unit would also never change anymore, and the unit would be forever useless - which seems much worse than even vanishingly small gradients. How do learning algorithms deal with that problem when they use ReLU?

Answer 1

Here is a paper that explains the issue. I'm quoting some part of it to make the issue clear.

The rectifier activation function allows a network to easily obtain sparse representations. For example, after uniform initialization of the weights, around 50% of hidden units continuous output values are real zeros, and this fraction can easily increase with sparsity-inducing regularization.

So rectifier activation function introduces sparsity effect on the network. Here are some advantages of sparsity from the same paper;

• Information disentangling. One of the claimed objectives of deep learning algorithms (Bengio,2009) is to disentangle the factors explaining the variations in the data. A dense representation is highly entangled because almost any change in the input modifies most of the entries in the representation vector. Instead, if a representation is both sparse and robust to small input changes, the set of non-zero features is almost always roughly conserved by small changes of the input.

• Efficient variable-size representation. Different inputs may contain different amounts of information and would be more conveniently represented using a variable-size data-structure, which is common in computer representations of information. Varying the number of active neurons allows a model to control the effective dimensionality of the representation for a given input and the required precision.

• Linear separability. Sparse representations are also more likely to be linearly separable, or more easily separable with less non-linear machinery, simply because the information is represented in a high-dimensional space. Besides, this can reflect the original data format. In text-related applications for instance, the original raw data is already very sparse.

• Distributed but sparse. Dense distributed representations are the richest representations, being potentially exponentially more efficient than purely local ones (Bengio, 2009). Sparse representations’ efficiency is still exponentially greater, with the power of the exponent being the number of non-zero features. They may represent a good trade-off with respect to the above criteria.

It also answers the question you've asked:

One may hypothesize that the hard saturation at 0 may hurt optimization by blocking gradient back-propagation. To evaluate the potential impact of this effect we also investigate the softplus activation: $ \text{softplus}(x) = \log(1 + e^x) $ (Dugas et al., 2001), a smooth version of the rectifying non-linearity. We lose the exact sparsity, but may hope to gain easier training. However, experimental results tend to contradict that hypothesis, suggesting that hard zeros can actually help supervised training. We hypothesize that the hard non-linearities do not hurt so long as the gradient can propagate along some paths, i.e., that some of the hidden units in each layer are non-zero With the credit and blame assigned to these ON units rather than distributed more evenly, we hypothesize that optimization is easier.

You can read the paper Deep Sparse Rectifier Neural Networks for more detail.

Answer 2

Here is a heuristic explanation:

• Each gradient update in backprop consists of a number of multiplied factors.
• The further you get towards the start of the network, the more of these factors are multiplied together to get the gradient update.
• Many of these factors are derivatives of the activation function of the neurons - the rest are weights, biases etc.
• Of these factors, the ones that intuitively matter are the weights, biases, etc. The activation function derivatives are more of a kind of tuning parameter, designed to get the gradient descent going in the right direction at the right kind of velocity.
• If you multiply a bunch of terms which are less than 1, they will tend towards zero the more terms you have. Hence vanishing gradient as you get further from the output layer if you have activation functions which have a slope < 1.
• If you multiply a bunch of terms which are greater than 1, they will tend towards infinity the more you have, hence exploding gradient as you get further from the output layer if you have activation functions which have a slope > 1.
• How about if we could, somehow, magically, get these terms contributed by the derivative of the activation functions to be 1. This intuitively means that all the contributions to the gradient updates come from the input to the problem and the model - the weights, inputs, biases - rather than some artefact of the activation function chosen.
• RELU has gradient 1 when output > 0, and zero otherwise.
• Hence multiplying a bunch of RELU derivatives together in the backprop equations has the nice property of being either 1 or zero - the update is either nothing, or takes contributions entirely from the other weights and biases.

You might think that it would be better to have a linear function, rather than flattening when x < 0. The idea here is that RELU generates sparse networks with a relatively small number of useful links, which has more biological plausibility, so the loss of a bunch of weights is actually helpful. Also, simulation of interesting functions with neural nets is only possible with some nonlinearity in the activation function. A linear activation function results in a linear output, which is not very interesting at all.

Answer 3

This is why it's probably a better idea to use PReLU, ELU, or other leaky ReLU-like activations which don't just die off to 0, but which fall to something like 0.1*x when x gets negative to keep learning. It seemed to me for a long time that ReLUs are history like sigmoid, though for some reason people still publish papers with these. Why? I don't know.

Dmytro Mishkin and other guys actually tested a network with plenty of different activation types, you should look into their findings on performance of different activation functions and other stuff. Some functions, like XOR, though, are better learnt with plain ReLU. Don't think about any neural stuff in dogma terms, because neural nets are very much work in progress. Nobody in the world actually knows and understands them well enough to tell the divine truth. Nobody. Try things out, make your own discoveries. Mind that using ReLU itself is a very recent development and for decades all the different PhD guys in the field have used over-complicated activation functions that we now can only laugh about. Too often "knowing" too much can get you bad results. It's important to understand that neural networks aren't an exact science. Nothing in maths says that neural networks will actually work as good as they do. It's heuristic. And so it's very malleable.

FYI even absolute-value activation gets good results on some problems, for example XOR-like problems. Different activation functions are better suited to different purposes. I tried Cifar-10 with abs() and it seemed to perform worse. Though, I can't say that "it is a worse activation function for visual recognition", because I'm not sure, for example, if my pre-initialization was optimal for it, etc. The very fact that it was learning relatively well amazed me.

Also, in real life, "derivatives" that you pass to the backprop don't necessarily have to match the actual mathematical derivatives.

I'd even go as far as to say we should ban calling them "derivatives" and start calling them something else, for example, error activation functions to not close our minds to possibilities of tinkering with them. You can actually, for example, use ReLU activation, but provide a 0.1, or something like that instead of 0 as a derivative for x<0. In a way, you then have a plain ReLU, but with neurons not being able to "die out of adaptability". I call this NecroRelu, because it's a ReLU that can't die. And in some cases (definitely not in most, though) that works better than plain LeakyReLU, which actually has 0.1 derivative at x<0 and better than usual ReLU. I don't think too many others have investigated such a function, though, this, or something similar might actually be a generally cool activation function that nobody considered just because they're too concentrated on maths.

As for what's generally used, for tanH(x) activation function it's a usual thing to pass 1 - x² instead of 1 - tanH(x)² as a derivative in order to calculate things faster.

Also, mind that ReLU isn't all that "obviously better" than, for example, TanH. TanH can probably be better in some cases. Just, so it seems, not in visual recognition. Though, ELU, for example, has a bit of sigmoid softness to it and it's one of the best known activation functions for visual recognition at the moment. I haven't really tried, but I bet one can set several groups with different activation functions on the same layer level to an advantage. Because, different logic is better described with different activation functions. And sometimes you probably need several types of evaluation.

Note that it's important to have an intialization that corresponds to the type of your activation function. Leaky ReLUs need other init that plain ReLUs, for example.

EDIT: Actually, standard ReLU seems less prone to overfitting vs leaky ones with modern architectures. At least in image recognition. It seems that if you are going for very high accuracy net with a huge load of parameters, it might be better to stick with plain ReLU vs leaky options. But, of course, test all of this by yourself. Maybe, some leaky stuff will work better if more regularization is given.

Answer 4

Let's consider the main recurrence relation that defines the back propagation of the error signal.

let $ {W_i}$ and ${b_i}$ be the weight matrix and bias vector of layer $\text{i}$, and ${f}$ be the activation function.

The activation vector ${h_i}$ of layer ${i}$ is calculated as follows:

${s_i} = {W_i}({h_{i-1}}) + {b_i} $

${h_i} = {f(s_i)}$

The error singal $\delta$ for layer ${i}$ is defined by:

${\delta_{i}} = {W_{i+1}({\delta_{i+1}}}\odot{f^{'}({s_i})})$

Where $\odot$ is elementwise multiplication of two vectors.

This recurrence relation is calculated for each layer in the network, and expresses the way the error signal is transferred from the output layer backwards. Now, if we take for example ${f}$ to be the tanh function, we have ${f^{'}({s_i})}=(1-h_i^2)$. Unless $h_i$ is exactly 1 or -1, this expression is a fraction between 0 to 1. Hence, each layer, the error signal is multiplied by a fraction, and becomes smaller and smaller: a vanishing gradient.

However, if we take ${f}=Relu=max(0,x)$, we have ${f^{'}}$ that is 1 for every neuron that has fired something, i.e. a neuron whose activation is nonzero (in numpy, this would be ${f^{'}} = \text{numpy.where}(h_i>0, 1, 0)$). In this case, the error signal is propagated fully to the next layer (it's multiplied by 1). Hence, even for a network with multiple layers, we don't encounter a vanishing gradient.

This equation also demonstrates the other problem characteristic to relu activation - dead neurons: if a given neuron happened to be initialized in a way that it doesn't fire for any input (its activation is zero), its gradient would also be zero, and hence it would never be activated.

Answer 5

Essentially, ReLUs only lead to vanishing gradients for inputs smaller than zero, while for other inputs they allow the gradient to pass through (look at plot of derivative of ReLU). This is unlike Sigmoid and Tanh, with can lead to gradient saturation for small and large values.

From Original ReLU paper:

Because of this linearity, gradients flow well on the active paths of neurons (there is no gradient vanishing effect due to activation non-linearities of sigmoid or tanh units), and mathematical investigation is easier.

Why not just have flowing gradients on both ends? Sparsity is beneficial:

One may hypothesize that the hard saturation at 0 may hurt optimization by blocking gradient back-propagation. To evaluate the potential impact of this effect we also investigate the softplus activation: softplus(x)=log(1+ex) (Dugas et al., 2001), a smooth version of the rectifying non-linearity. We lose the exact sparsity, but may hope to gain easier training. However, experimental results tend to contradict that hypothesis, suggesting that hard zeros can actually help supervised training. We hypothesize that the hard non-linearities do not hurt so long as the gradient can propagate along some paths, i.e., that some of the hidden units in each layer are non-zero With the credit and blame assigned to these ON units rather than distributed more evenly, we hypothesize that optimization is easier.

Paper