Numeric Gradient Checking: How close is close enough?

Question

I made a convolutional neural network and I wanted to check that my gradients are being calculated correctly using numeric gradient checking. The question is, how close is close enough?

My checking function just spits out the calculated derivative, the numerically approximated derivative, the difference between the two, and whether or not the two values have the same sign (one being positive and the other being negative is a big no-no) for each weight.

The main concern I have is that for all the fully connected layers and all the convolutional layers except the first one, the differences look similar - the first 9-13 characters of the two numbers will match. That sounds good enough, right? But for weights of the first convolutional layer, sometimes I get up to 12 decimal places to match but it can also be as low as just 3. Is that enough, or could there be a possible error?

One good thing to note is the sign of the two values is always matching which is good, so the network will always make moves in the right direction, even if the magnitude of the movement is a bit off. But that's the question... is there a chance that it is off?

Answer 1

The closest I have seen to addressing this was in the Stanford UFLDL tutorial within the softmax regression section. Copying the key statement:

The norm of the difference between the numerical gradient and your analytical gradient should be small, on the order of $10^{-9}$.

In python the code would look something like this:

norm(gradients - numericalGradients)/norm(gradients + numericalGradients)

where gradients are you results from the derivative and numericalGradients are the approximated gradients.

Answer 2

Background Theory that's helpful

One small fact that you can use to help understand whether a numeric derivative is correctly calculated or not is the Cauchy Remainder of the Taylor expansion. That is,

$f(x + h) = f(x) + hf'(x) + \frac{h^2}{2}f''(\xi)$ for some $\xi \in [x, x+ h]$

This is helpful, because you've probably approximated your first derivative by

$f'(x)\approx \frac{f(x+h) - f(x-h)}{2h}$

with some small $h$ (I typically use $10^{-4}$, but I'm sure some day I'll run across a case where that's not appropriate).

After a little algebra, we can use the Cauchy remainder to see that our numeric approximation theoretically should be within $h f''(\xi), \xi \in [x-h, x+h]$ of $f'(x)$.

In fact, you can actually bound it by $h (f''(\xi_1) - f''(\xi_2) )$, where $\xi_1 \in [x-h, x]$ and $\xi_2 \in [x, x+h]$...which is equivalent to $h^2f'''(\xi)$, $\xi \in [x-h, x+h]$.

Problems in Practice

Okay, we have nice theory bounding the error of the numeric derivative. But there are two holes in directly trying to use those results:

1.) We don't know $f'''(x)$ (and probably don't want to spend the time approximating it)

2.) as $h \rightarrow 0$, $\frac{f(x+h) - f(x-h)}{2h}$ suffers from numeric instability

So using what we know from earlier the way I check my analytic derivatives (which might not be the best way) is that I write the numeric derivative function as a function of $h$. If I can't tell whether the difference between the numeric and analytic derivatives is due to a coding mistake or just numeric approximation, I can reduce $h$ and see if my numeric derivative approaches my analytic derivative before suffering from numeric instability (when this happens, your numeric approximations will become less consistent as $h$ gets smaller). Note that $f'''(\xi)$ term should be disappearing quadratically, so if my error is about $0.01$ with $h = 10^{-4}$, it should be around $0.0001$ with $h = 10^{-5}$ assuming numeric instability has not kicked in yet.

Unfortunately, there's no hard and fast guideline for always determining these things; it's very dependent on how stable the function is (and I mean both in terms in numeric stability and higher derivatives). But in my experiences, I've never seen a case where the error from $h^2 f'''(\xi)$ was not definitively going to 0 (i.e. using $h = 10^{-4}$ gave virtually the same answer as $h = 10^{-5}$) by the time the numeric instability from $h \rightarrow 0$ kicked in.

Answer 3

Please refer to this tutorial http://cs231n.github.io/neural-networks-3/#ensemble. The "Gradient Check" section is very detailed and helpful.

As is suggested by gung, I include the main points of this link:

• Use $\frac{f(w+h)-f(w-h)}{2h}$ approximation, where $h\sim 10^{-5}$.

• Monitor the fraction of $\frac{|f_a'(w)-f_n'(w)|}{max(|f_a'(w)|,|f_n'(w)|)}$, where $f'_a(w)$ is the analytical gradient and $f'_n(w)$ is the numerically approximated gradient. Usually, the preferred range of this fraction should $<10^{-2}$.

• Use double precision instead of float.

• Mind of kink(s) in activation functions, e.g., $x=0$ when one uses ReLU. When there is kink(s), one needs to monitor the values of $x-h$ and $x+h$. If these two values are on two sides of a kink, one should exclude this gradient check.

• Use few datapoints.

• Do not do gradient checking at the very beginning stage of the training process.

• First check model without regularization and then with it.

• Turn off dropout and inverted dropout when doing gradient checking.

• Only check randomly few dimensions.