2
$\begingroup$

In Goodfellow et al.'s Deep Learning, the authors write, "rectified linear units use the activation function $g(z) = \max\{0,z\}$... The gradients are not only large but consistent" (187). What does it mean for a gradient to be consistent? I always understood consistency to mean that for an estimator of a parameter, the estimator converges in probability to the true parameter value as the number of data points goes to infinity. How does this definition relate to the gradient? Does this mean that if you use SGD, the stochastic gradient converges to the true gradient as the number of data points goes to infinity, and if so, isn't this a general property of SGD rather than of ReLU?

$\endgroup$
1
  • 1
    $\begingroup$ I think he means that the gradient is either 1 or 0. $\endgroup$ – Sycorax Aug 25 '17 at 0:22
2
$\begingroup$

The word "consistent" here doesn't have the technical meaning of the consistency of a statistical estimator. My read is that "consistent" has plan English meaning.

When $x$ is positive, the gradient for $\text{ReLU}(x)$ is 1, otherwise it's 0. In other words, I think he means that the gradient is either 0 or 1, and not any other value. This interpretation seems likely given the complete context of the passage:

These [ReLU] units are easy to optimize because they are so similar to linear units. The only difference between a linear unit and a rectified linear unit is that a rectified linear unit outputs zero across half its domain. This makes the derivatives through a rectified linear unit remain large whenever the unit is active. The gradients are not only large but also consistent. The second derivative of the rectifying operation is 0 almost everywhere, and the derivative of the rectifying operation is 1 everywhere that the unit is active.

We can compare this to something like $\tanh(x)$, which has a gradient that varies in $x$, so in that sense the $\tanh$ gradient is not "consistent," but always changing.

$\endgroup$
0
0
$\begingroup$

He's referring to how ReLU isn't bounded from above, which allows the training process to handle all (positive) activation of a neuron according to how activated it is, as opposed to tanh for example which provides 0 gradient when "saturated" by large values. ReLU is good for convolutional neural networks for example because the process looks for the presence of features as it scans across an image, which manifests as large outputs of the matrix convolution operation. If we used a bounded activation function, we'd be discarding all the information that tells us how "large" of a presence some feature had vs another arbitrary presence.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.