Numerical check of gradient in neural network I am trying to check if my implementation of backpropogation is correct by checking the calculated gradients with the numeric gradient.
I am testing it on a very simple linear network (i.e. no sigmoid on the output activation) as shown below:

There are two input neurons and one output neuron. The weights have the values [0.34, 0.56] and the bias is 0.54
The loss function used is the squared loss i.e.
$$
\frac{1}{2}(y - output)^2
$$
With an input of [1, 0], working out the gradients w.r.t the weights (dw) and bias (db) using the chain rule (backpropogation) gives me:
dw = [0.7744, 0] and db = [0.7744] which is exactly what my backprop implementation gives.
However, to calculate the numerical gradient my understanding is that I should compute
$g(\theta) \approx \frac{J(\theta + \epsilon) - J(\theta - \epsilon)}{2\epsilon}$ where $J$ is the network output.
Using the example input of [1,0] and perturbing the 0.34 weight with $+/- \epsilon$ the numeric gradient will always be $1$, regardless of what I use for $\epsilon$.  This is quite far away from my analytic gradient. 
Am I doing something wrong?
Thanks
 A: What is $J(\theta)$?  
If it is the output (without activation functions) directly, 
$J(\theta)=ax_1+bx_2+c-y$.  
Then $\frac{\partial J}{\partial a} = x_1=1$ is correct.
If it is the squared loss,   
$J(\theta)=\frac{1}{2}(ax_1+bx_2+c-y)^2$,  
Then,  
$\frac{\partial J}{\partial a} = (ax_1+bx_2+c-y)x_1=0.34+0.54-y$,  
which might be the value 0.7744 you got using backpropogation.  
Perhaps you might have used different $J(\theta)$ for backpropogation and numerical check.
Hope that helps.
A: The error here is simply that you should take the derivative of your loss function,
$$
E(\theta) = \frac{1}{2}\, (\, target - output\,(\,input, \, \theta)\,)^2\,,
$$
and not the one of your network output $J$. So you should calculate
$$g(\theta) \approx \frac{E(\theta + \epsilon) - E(\theta - \epsilon)}{2\epsilon}$$
instead.
Rationale: you want to minimize $E$ with respect to the parameter $\theta$ (the neural network weights). So you always go a step into the direction in which the decrease of $E$ is maximal, which is given by $-\frac{\partial E}{\partial \theta_{ij}}$ (where $\theta_{ij}$ is a network weight).
Bishop gives a short section on numerical evaluation of the derivative and the hessian in neural networks in his book "pattern recognition and machine learning".
