Why do we use rectified linear units (ReLU) with neural networks? How does that improve neural network?

Why do we say that ReLU is an activation function? Isn't softmax activation function for neural networks? I am guessing that we use both, ReLU and softmax, like this:

neuron 1 with softmax output ----> ReLU on the output of neuron 1, which is
input of neuron 2 ---> neuron 2 with softmax output --> ...

so that the input of neuron 2 is basically ReLU(softmax(x1)). Is this correct?


The ReLU function is $f(x)=\max(0, x).$ Usually this is applied element-wise to the output of some other function, such as a matrix-vector product. In MLP usages, rectifier units replace all other activation functions except perhaps the readout layer. But I suppose you could mix-and-match them if you'd like.

One way ReLUs improve neural networks is by speeding up training. The gradient computation is very simple (either 0 or 1 depending on the sign of $x$). Also, the computational step of a ReLU is easy: any negative elements are set to 0.0 -- no exponentials, no multiplication or division operations.

Gradients of logistic and hyperbolic tangent networks are smaller than the positive portion of the ReLU. This means that the positive portion is updated more rapidly as training progresses. However, this comes at a cost. The 0 gradient on the left-hand side is has its own problem, called "dead neurons," in which a gradient update sets the incoming values to a ReLU such that the output is always zero; modified ReLU units such as ELU (or Leaky ReLU, or PReLU, etc.) can ameliorate this.

$\frac{d}{dx}\text{ReLU}(x)=1\forall x > 0$ . By contrast, the gradient of a sigmoid unit is at most $0.25$; on the other hand, $\tanh$ fares better for inputs in a region near 0 since $0.25 < \frac{d}{dx}\tanh(x) \le 1 \forall x \in [-1.31, 1.31]$ (approximately).

  • $\begingroup$ @aginensky You can ask questions by clicking the Ask Question button at the top of the page. $\endgroup$ – Sycorax says Reinstate Monica Jun 24 '19 at 18:28
  • $\begingroup$ I see no evidence that I wanted to ask a question or that I participated in this page. Frankly I'm amazed at how well ReLU works, but I've stopped questioning it :). $\endgroup$ – meh Jun 24 '19 at 20:14
  • $\begingroup$ @aginensky It appears that the comment was removed in the interim. $\endgroup$ – Sycorax says Reinstate Monica Jun 24 '19 at 20:22
  • $\begingroup$ The comment was not removed by me nor was I informed. I've stopped answering questions and I guess this means I'm done with commenting too. $\endgroup$ – meh Jun 24 '19 at 21:06
  • $\begingroup$ @aginensky I don't know why this would cause you to stop commenting. If you have any questions about comments and moderation, you could ask a question in meta.stats.SE. $\endgroup$ – Sycorax says Reinstate Monica Jun 25 '19 at 12:29

One important thing to point out is that ReLU is idempotent. Given that ReLU is $\rho(x) = \max(0, x)$, it's easy to see that $\rho \circ \rho \circ \rho \circ \dots \circ \rho = \rho$ is true for any finite composition. This property is very important for deep neural networks, because each layer in the network applies a nonlinearity. Now, let's apply two sigmoid-family functions to the same input repeatedly 1-3 times:

enter image description here

You can immediately see that sigmoid functions "squash" their inputs resulting in the vanishing gradient problem: derivatives approach zero as $n$ (the number of repeated applications) approaches infinity.


ReLU is the max function(x,0) with input x e.g. matrix from a convolved image. ReLU then sets all negative values in the matrix x to zero and all other values are kept constant.

ReLU is computed after the convolution and therefore a nonlinear activation function like tanh or sigmoid.

Softmax is a classifier at the end of the neural network. That is logistic regression to regularize outputs to values between 0 and 1. (Alternative here is a SVM classifier).

CNN Forward Pass e.g.: input->conv->ReLU->Pool->conv->ReLU->Pool->FC->softmax

  • 8
    $\begingroup$ Downvoting. This a very bad answer! Softmax is not a classifier! It is a function that normalizes (scales) the outputs to the range [0,1] and ensures they sum up to 1. Logistic regression does not "regularize" anything! The sentence "ReLU is computed after the convolution and therefore a nonlinear activation function like tanh or sigmoid." lacks a verb, or sense. $\endgroup$ – Jan Kukacka Dec 25 '17 at 20:36
  • 2
    $\begingroup$ The answer is not that bad. The sentence without the verb must be "ReLU is computed after the convolution and IS therefore a nonlinear activation function like tanh or sigmoid." Thinking of softmax as a classifier makes sense too. It can be seen as a probabilistic classifier that assigns a probability to each class. It "regularizes"/"normalizes" the outputs to the [0,1] interval. $\endgroup$ – user118967 Nov 18 '18 at 3:54

ReLU is a literal switch. With an electrical switch 1 volt in gives 1 volt out, n volts in gives n volts out when on. On/Off when you decide to switch at zero gives exactly the same graph as ReLU. The weighted sum (dot product) of a number of weighted sums is still a linear system. For a particular input the ReLU switches are individually on or off. That results in a particular linear projection from the input to the output, as various weighted sums of weighted sum of ... are connected together by the switches. For a particular input and a particular output neuron there is a compound system of weighted sums that actually can be summarized to a single effective weighted sum. Since ReLU switches state at zero there are no sudden discontinuities in the output for gradual changes in the input.

There are other numerically efficient weighted sum (dot product) algorithms around like the FFT and Walsh Hadamard transform. There is no reason you can't incorporate those into an ReLU based neural network and benefit from the computational gains. (eg. Fixed filter bank neural networks.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.