Is gradient checking useless in high dimensional setting? Is gradient checking (finite difference for numerical gradient to check if analytical gradient is correct) useless in high dimensional setting (say 100K parameters in a deep neural network)? 
Here is my reason.


*

*First, computing numerical gradient is extremely slow. I experimented with R numderiv library, it will not give me any results in couple of hours with even 10K paramters.

*Second, if we get norm between two very long vectors $\|a-b\|$, it will still be reasonable big (say around $1 \times 10^{-3}$)
Am I right?
 A: Use the infinity norm (i.e., maximum absolute deviation across all components), not the 2 norm.  Then it will essentially be independent of number of components (dimension).  If you do gradient check on at least some components, that's better than nothing. And you do it once, when you first run your model, not every time you run it.
You know why gradient checking is even around?  Keep in mind "everyone" knows it (finite differences) can take a long time in high dimensions.  Because people get the gradient wrong so often, that's why. Even the great me has gotten it wrong. But maybe you will be one of the few who never do, so congratulations on that.
A: This question seems misguided to me. Gradient checking is done to check if the gradient has been coded correctly (for perspective: I think I code about 5% of my gradients correctly the first time). Once you've checked that your code works correctly, it's not done any more; the analytic gradient is used. 
As such, size of the model should generally not be a concern. When coding things up, you should always start with a test model that's small enough for you to keep track of. The speed of gradient checking will not be a problem there! Similarly, you may use random data to check things, but generally speaking, there's no need to scale up your random data in the test runs. 
So unless you are building a general model that always needs to be huge, the issue of scaling numeric gradients for gradient checking is somewhat irrelevant: you don't do gradient checking once you've verified that your analytic gradient is working properly. This verification is usually done on small toy models. 
A: If you directly do gradient check on a network of $n$ parameters, you'll need $O(n)$ forward passes. The time complexity of a forward pass is $O(n)$ in terms of length growth and $O(n^2)$ in terms of width growth. So the total time complexity will be in between $O(n^2)$ and $O(n^3)$, clearly we don't want $n$ to be something like 100k in such case.
As GeoMatt22 commented, unit tests and integrated tests should be enough. 
For example a network made up by stacking many convolutional layers, we can first do numerical check for a single convolutional layer, then do numerical check on stacking two or more layers, after the tests we'll just build a network with whatever architecture.
