In Goodfellow et al.'s Deep Learning, the authors write, "rectified linear units use the activation function $g(z) = \max\{0,z\}$... The gradients are not only large but consistent" (187). What does it mean for a gradient to be consistent? I always understood consistency to mean that for an estimator of a parameter, the estimator converges in probability to the true parameter value as the number of data points goes to infinity. How does this definition relate to the gradient? Does this mean that if you use SGD, the stochastic gradient converges to the true gradient as the number of data points goes to infinity, and if so, isn't this a general property of SGD rather than of ReLU?

• I think he means that the gradient is either 1 or 0. – Sycorax Aug 25 '17 at 0:22

When $$x$$ is positive, the gradient for $$\text{ReLU}(x)$$ is 1, otherwise it's 0. In other words, I think he means that the gradient is either 0 or 1, and not any other value. This interpretation seems likely given the complete context of the passage:
We can compare this to something like $$\tanh(x)$$, which has a gradient that varies in $$x$$, so in that sense the $$\tanh$$ gradient is not "consistent," but always changing.