# Why do nonlinearities in deep neural nets give rise to very high derivatives?

In the book "Deep Learning" by Goodfellow, Bengio, and Courville, I do not understand the following statement about why nonlinearities in deep neural nets give rise to very high derivatives:

The objective function for highly nonlinear deep neural networks or for recurrent neural networks often contains sharp nonlinearities in parameter space resulting from the multiplication of several parameters. These nonlinearities give rise to very high derivatives in some places. When the parameters get close to such a cliff region, a gradient descent update can catapult the parameters very far, possibly losing most of the optimization work that has been done.

• Isn't this related to the exploding gradient problem? Sep 6, 2018 at 12:29

As mentioned in Digio's comment, this is a description of the exploding gradient problem. If you are familiar with Newton's method from calculus, we can look at a simple example in the single variable case. Updates are proportional to the value of the derivative of the function: $x_{n+1} = x_{n} - \frac{f(x_{n})}{f'(x_{n})}$ If the value of the derivative at $x_{n}$ is huge, $\Delta x = x_{n+1} - x_{n}$ may diverge to infinity - i.e., explode. Here is an image of this same phenomenon occurring with a multivariate function: 