2
$\begingroup$

Is gradient checking (finite difference for numerical gradient to check if analytical gradient is correct) useless in high dimensional setting (say 100K parameters in a deep neural network)?

Here is my reason.

  • First, computing numerical gradient is extremely slow. I experimented with R numderiv library, it will not give me any results in couple of hours with even 10K paramters.

  • Second, if we get norm between two very long vectors $\|a-b\|$, it will still be reasonable big (say around $1 \times 10^{-3}$)

Am I right?

$\endgroup$
  • 1
    $\begingroup$ Can you clarify the term "gradient checking"? Do you mean verifying an analytical gradient code by checking against a finite difference version? (If yes, then this question might be better suited to scicomp.stackexchange.com ) $\endgroup$ – GeoMatt22 Sep 23 '16 at 4:08
  • 2
    $\begingroup$ You may also check Ian Goodfellow's Deep Learning book here, where he suggests a strategy using "random projections" (section 11.5, page 439) to give confidence with fewer forward evaluations. $\endgroup$ – GeoMatt22 Sep 23 '16 at 4:15
  • 2
    $\begingroup$ Though I have not done any deep learning, my instinct says that for pure code-verification the high-D setting is unlikely to be the best testing environment. In particular, all the relevant unit tests as well as the composite-architecture integrated tests should be very do-able using a combination of modular functions and/or small problems. For example, if you greatly decrease the nodes/layer but keep layer/activation ty[es and depth, probably you will catch most implementation problems. $\endgroup$ – GeoMatt22 Sep 23 '16 at 5:43
1
$\begingroup$

If you directly do gradient check on a network of $n$ parameters, you'll need $O(n)$ forward passes. The time complexity of a forward pass is $O(n)$ in terms of length growth and $O(n^2)$ in terms of width growth. So the total time complexity will be in between $O(n^2)$ and $O(n^3)$, clearly we don't want $n$ to be something like 100k in such case.

As GeoMatt22 commented, unit tests and integrated tests should be enough.

For example a network made up by stacking many convolutional layers, we can first do numerical check for a single convolutional layer, then do numerical check on stacking two or more layers, after the tests we'll just build a network with whatever architecture.

$\endgroup$
  • 1
    $\begingroup$ Apply the comparison for a few random vectors. There is no need to compute all $n^2$ entries to have high confidence an analytic gradient code is correct. This comparison is routine in nonlinear solvers libraries like PETSc. $\endgroup$ – Jed May 1 '17 at 20:17
4
$\begingroup$

Use the infinity norm (i.e., maximum absolute deviation across all components), not the 2 norm. Then it will essentially be independent of number of components (dimension). If you do gradient check on at least some components, that's better than nothing. And you do it once, when you first run your model, not every time you run it.

You know why gradient checking is even around? Keep in mind "everyone" knows it (finite differences) can take a long time in high dimensions. Because people get the gradient wrong so often, that's why. Even the great me has gotten it wrong. But maybe you will be one of the few who never do, so congratulations on that.

$\endgroup$
3
$\begingroup$

This question seems misguided to me. Gradient checking is done to check if the gradient has been coded correctly (for perspective: I think I code about 5% of my gradients correctly the first time). Once you've checked that your code works correctly, it's not done any more; the analytic gradient is used.

As such, size of the model should generally not be a concern. When coding things up, you should always start with a test model that's small enough for you to keep track of. The speed of gradient checking will not be a problem there! Similarly, you may use random data to check things, but generally speaking, there's no need to scale up your random data in the test runs.

So unless you are building a general model that always needs to be huge, the issue of scaling numeric gradients for gradient checking is somewhat irrelevant: you don't do gradient checking once you've verified that your analytic gradient is working properly. This verification is usually done on small toy models.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.