In a network, we find gradients of the error function w.r.t each of the parameters used in the network.
We then update the weights say, using vanilla Gradient Descent.
If the computed gradients, do not match the analytical gradient value by a small fraction, can it still converge?
The reason I'm asking: I wrote a neural network from scratch using numpy. It converged on MNIST (so there was a lot of data). But then I checked my computed gradients with autograd, and I can see they're not exactly the same.
Why did the network converge then?
- What are the cases where slightly wrong gradients, can still give you good results?
- Under what circumstances will this break?
I eventually fixed my gradient computation, and then it converged faster and better. But why did it converge with wrong gradients in the first place?
Note: Magnitude of "slightly incorrect" for example, is like this : Computed gradient = 0.0014, Correct Gradient = -0.0008. (just to give a baseline)