# Why do we use ReLU in neural networks and how do we use it?

Why do we use rectified linear units (ReLU) with neural networks? How does that improve neural network?

Why do we say that ReLU is an activation function? Isn't softmax activation function for neural networks? I am guessing that we use both, ReLU and softmax, like this:

neuron 1 with softmax output ----> ReLU on the output of neuron 1, which is
input of neuron 2 ---> neuron 2 with softmax output --> ...

so that the input of neuron 2 is basically ReLU(softmax(x1)). Is this correct?

The ReLU function is $$f(x)=\max(0, x).$$ Usually this is applied element-wise to the output of some other function, such as a matrix-vector product. In MLP usages, rectifier units replace all other activation functions except perhaps the readout layer. But I suppose you could mix-and-match them if you'd like.

One way ReLUs improve neural networks is by speeding up training. The gradient computation is very simple (either 0 or 1 depending on the sign of $$x$$). Also, the computational step of a ReLU is easy: any negative elements are set to 0.0 -- no exponentials, no multiplication or division operations.

Gradients of logistic and hyperbolic tangent networks are smaller than the positive portion of the ReLU. This means that the positive portion is updated more rapidly as training progresses. However, this comes at a cost. The 0 gradient on the left-hand side is has its own problem, called "dead neurons," in which a gradient update sets the incoming values to a ReLU such that the output is always zero; modified ReLU units such as ELU (or Leaky ReLU, or PReLU, etc.) can ameliorate this.

$$\frac{d}{dx}\text{ReLU}(x)=1\forall x > 0$$ . By contrast, the gradient of a sigmoid unit is at most $$0.25$$; on the other hand, $$\tanh$$ fares better for inputs in a region near 0 since $$0.25 < \frac{d}{dx}\tanh(x) \le 1 \forall x \in [-1.31, 1.31]$$ (approximately).

• @aginensky You can ask questions by clicking the Ask Question button at the top of the page. – Sycorax Jun 24 at 18:28
• I see no evidence that I wanted to ask a question or that I participated in this page. Frankly I'm amazed at how well ReLU works, but I've stopped questioning it :). – aginensky Jun 24 at 20:14
• @aginensky It appears that the comment was removed in the interim. – Sycorax Jun 24 at 20:22
• The comment was not removed by me nor was I informed. I've stopped answering questions and I guess this means I'm done with commenting too. – aginensky Jun 24 at 21:06
• @aginensky I don't know why this would cause you to stop commenting. If you have any questions about comments and moderation, you could ask a question in meta.stats.SE. – Sycorax Jun 25 at 12:29

One important thing to point out is that ReLU is idempotent. Given that ReLU is $$\rho(x) = \max(0, x)$$, it's easy to see that $$\rho \circ \rho \circ \rho \circ \dots \circ \rho = \rho$$ is true for any finite composition. This property is very important for deep neural networks, because each layer in the network applies a nonlinearity. Now, let's apply two sigmoid-family functions to the same input repeatedly 1-3 times:

You can immediately see that sigmoid functions "squash" their inputs resulting in the vanishing gradient problem: derivatives approach zero as $$n$$ (the number of repeated applications) approaches infinity.

ReLU is the max function(x,0) with input x e.g. matrix from a convolved image. ReLU then sets all negative values in the matrix x to zero and all other values are kept constant.

ReLU is computed after the convolution and therefore a nonlinear activation function like tanh or sigmoid.

Softmax is a classifier at the end of the neural network. That is logistic regression to regularize outputs to values between 0 and 1. (Alternative here is a SVM classifier).

CNN Forward Pass e.g.: input->conv->ReLU->Pool->conv->ReLU->Pool->FC->softmax

• Downvoting. This a very bad answer! Softmax is not a classifier! It is a function that normalizes (scales) the outputs to the range [0,1] and ensures they sum up to 1. Logistic regression does not "regularize" anything! The sentence "ReLU is computed after the convolution and therefore a nonlinear activation function like tanh or sigmoid." lacks a verb, or sense. – Jan Kukacka Dec 25 '17 at 20:36
• The answer is not that bad. The sentence without the verb must be "ReLU is computed after the convolution and IS therefore a nonlinear activation function like tanh or sigmoid." Thinking of softmax as a classifier makes sense too. It can be seen as a probabilistic classifier that assigns a probability to each class. It "regularizes"/"normalizes" the outputs to the [0,1] interval. – user118967 Nov 18 '18 at 3:54