Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Why do we use rectified linear units (ReLU) with neural networks? How does that improve neural network?
Why do we say that ReLU is an activation function? Isn't softmax activation function for neural networks? I am guessing that we use both, ReLU and softmax, like this:
neuron 1 with softmax output ----> ReLU on the output of neuron 1, which is
input of neuron 2 ---> neuron 2 with softmax output --> ...
so that the input of neuron 2 is basically ReLU(softmax(x1)). Is this correct?
The ReLU function is $f(x)=\max(0, x).$ Usually this is applied element-wise to the output of some other function, such as a matrix-vector product. In MLP usages, rectifier units replace all other activation functions except perhaps the readout layer. But I suppose you could mix-and-match them if you'd like.
One way ReLUs improve neural networks is by speeding up training. The gradient computation is very simple (either 0 or 1 depending on the sign of $x$). Also, the computational step of a ReLU is easy: any negative elements are set to 0.0 -- no exponentials, no multiplication or division operations.
Gradients of logistic and hyperbolic tangent networks are smaller than the positive portion of the ReLU. This means that the positive portion is updated more rapidly as training progresses. However, this comes at a cost. The 0 gradient on the left-hand side is has its own problem, called "dead neurons," in which a gradient update sets the incoming values to a ReLU such that the output is always zero; modified ReLU units such as ELU (or Leaky ReLU, or PReLU, etc.) can ameliorate this.
$\frac{d}{dx}\text{ReLU}(x)=1\forall x > 0$ . By contrast, the gradient of a sigmoid unit is at most $0.25$; on the other hand, $\tanh$ fares better for inputs in a region near 0 since $0.25 < \frac{d}{dx}\tanh(x) \le 1 \forall x \in [-1.31, 1.31]$ (approximately).
One important thing to point out is that ReLU is idempotent. Given that ReLU is $\rho(x) = \max(0, x)$, it's easy to see that $\rho \circ \rho \circ \rho \circ \dots \circ \rho = \rho$ is true for any finite composition. This property is very important for deep neural networks, because each layer in the network applies a nonlinearity. Now, let's apply two sigmoid-family functions to the same input repeatedly 1-3 times:
You can immediately see that sigmoid functions "squash" their inputs resulting in the vanishing gradient problem: derivatives approach zero as $n$ (the number of repeated applications) approaches infinity.
Why do we use ReLUs? We use ReLUs for the same reason we use any other non-linear activation function: To achieve a non-linear transformation of the data.
Why do we need non-linear transformations? We apply non-linear transformations in the hope that the transformed data will be (close to) linear (for regression) or (close to) linearly separable (for classification). Drawing a linear function through non-linearly transformed data is equivalent to drawing a non-linear function through original data.
Why are ReLUs better than other activation functions? They are simple, fast to compute, and don't suffer from vanishing gradients, like sigmoid functions (logistic, tanh, erf, and similar). The simplicity of implementation makes them suitable for use on GPUs, which are very common today due to being optimised for matrix operations (which are also needed for 3D graphics).
Why do we need matrix operations in neural networks?: It's a compact and computationally efficient way of propagating the signals between the layers (multiplying the output of the previous layer with the weight matrix).
Isn't softmax activation function for neural networks? Softmax is not really an activation function of a single neuron, but a way of normalising outputs of multiple neurons. It is usually used in the output layer, to enforce the sum of outputs to be one, so that they can be interpreted as probabilities. You could also use it in hidden layers, to enforce the outputs to be in a limited range, but other approaches, like batch normalisation, are better suited for that purpose.
P.S. (1) ReLU stands for "rectified linear unit", so, strictly speaking, it is a neuron with a (half-wave) rectified-linear activation function. But people usually mean the activation function when they talk about ReLUs.
P.S. (2) Passing the output of softmax to a ReLU doesn't have any effect because softmax produces only non-negative values, in range $[0, 1]$, where ReLU acts as identity function, i.e. doesn't change them.
ReLU is a literal switch. With an electrical switch 1 volt in gives 1 volt out, n volts in gives n volts out when on. On/Off when you decide to switch at zero gives exactly the same graph as ReLU. The weighted sum (dot product) of a number of weighted sums is still a linear system. For a particular input the ReLU switches are individually on or off. That results in a particular linear projection from the input to the output, as various weighted sums of weighted sum of ... are connected together by the switches. For a particular input and a particular output neuron there is a compound system of weighted sums that actually can be summarized to a single effective weighted sum. Since ReLU switches state at zero there are no sudden discontinuities in the output for gradual changes in the input.
There are other numerically efficient weighted sum (dot product) algorithms around like the FFT and Walsh Hadamard transform. There is no reason you can't incorporate those into an ReLU based neural network and benefit from the computational gains. (eg. Fixed filter bank neural networks.)
ReLU is the max function(x,0) with input x e.g. matrix from a convolved image. ReLU then sets all negative values in the matrix x to zero and all other values are kept constant.
ReLU is computed after the convolution and is a nonlinear activation function like tanh or sigmoid.
Softmax is a classifier at the end of the neural network. That is logistic regression to normalize outputs to values between 0 and 1. (Alternative here is a SVM classifier).
CNN Forward Pass e.g.: input->conv->ReLU->Pool->conv->ReLU->Pool->FC->softmax
ReLU is probably one of the simplest nonlinear function possible. A step function is simpler. However, a step function has the first derivative (gradient) zero everywhere but in one point, at which it has an infinite gradient. ReLU has a finite derivative (gradient) everywhere. It has an infinite second derivative in one point.
The feed forward networks are trained by looking for a zero gradient. The important thing here is that there's a lot of first derivatives to calculate in a large net's backpropagation routine, and it helps that they are fast to compute like ReLU. The second is that unlike step function, ReLU's gradients are always finite and they're not trivial zeros almost everywhere. Finally, we need nonlinear activations for the deep learning net to work well, but that's different subject.
