1
$\begingroup$

In Goodfellow et al.'s Deep Learning, the authors write, "rectified linear units use the activation function $g(z) = \max\{0,z\}$... The gradients are not only large but consistent" (187). What does it mean for a gradient to be consistent? I always understood consistency to mean that for an estimator of a parameter, the estimator converges in probability to the true parameter value as the number of data points goes to infinity. How does this definition relate to the gradient? Does this mean that if you use SGD, the stochastic gradient converges to the true gradient as the number of data points goes to infinity, and if so, isn't this a general property of SGD rather than of ReLU?

$\endgroup$
  • 1
    $\begingroup$ I think he means that the gradient is either 1 or 0. $\endgroup$ – Reinstate Monica Aug 25 '17 at 0:22
0
$\begingroup$

He's referring to how ReLU isn't bounded from above, which allows the training process to handle all (positive) activation of a neuron according to how activated it is, as opposed to tanh for example which provides 0 gradient when "saturated" by large values. ReLU is good for convolutional neural networks for example because the process looks for the presence of features as it scans across an image, which manifests as large outputs of the matrix convolution operation. If we used a bounded activation function, we'd be discarding all the information that tells us how "large" of a presence some feature had vs another arbitrary presence.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.