I am trying to check if my implementation of backpropogation is correct by checking the calculated gradients with the numeric gradient.

I am testing it on a very simple linear network (i.e. no sigmoid on the output activation) as shown below:

enter image description here

There are two input neurons and one output neuron. The weights have the values [0.34, 0.56] and the bias is 0.54

The loss function used is the squared loss i.e.

$$ \frac{1}{2}(y - output)^2 $$

With an input of [1, 0], working out the gradients w.r.t the weights (dw) and bias (db) using the chain rule (backpropogation) gives me: dw = [0.7744, 0] and db = [0.7744] which is exactly what my backprop implementation gives.

However, to calculate the numerical gradient my understanding is that I should compute $g(\theta) \approx \frac{J(\theta + \epsilon) - J(\theta - \epsilon)}{2\epsilon}$ where $J$ is the network output.

Using the example input of [1,0] and perturbing the 0.34 weight with $+/- \epsilon$ the numeric gradient will always be $1$, regardless of what I use for $\epsilon$. This is quite far away from my analytic gradient.

Am I doing something wrong?



What is $J(\theta)$?

If it is the output (without activation functions) directly,


Then $\frac{\partial J}{\partial a} = x_1=1$ is correct.

If it is the squared loss,



$\frac{\partial J}{\partial a} = (ax_1+bx_2+c-y)x_1=0.34+0.54-y$,

which might be the value 0.7744 you got using backpropogation.

Perhaps you might have used different $J(\theta)$ for backpropogation and numerical check. Hope that helps.

| cite | improve this answer | |
  • $\begingroup$ Hi, sorry forgot to mention this (have edited the question) I am using squared loss i.e. 0.5 * (y - output)^2 $\endgroup$ – Aly Dec 30 '15 at 11:11

The error here is simply that you should take the derivative of your loss function,

$$ E(\theta) = \frac{1}{2}\, (\, target - output\,(\,input, \, \theta)\,)^2\,, $$

and not the one of your network output $J$. So you should calculate

$$g(\theta) \approx \frac{E(\theta + \epsilon) - E(\theta - \epsilon)}{2\epsilon}$$


Rationale: you want to minimize $E$ with respect to the parameter $\theta$ (the neural network weights). So you always go a step into the direction in which the decrease of $E$ is maximal, which is given by $-\frac{\partial E}{\partial \theta_{ij}}$ (where $\theta_{ij}$ is a network weight).

Bishop gives a short section on numerical evaluation of the derivative and the hessian in neural networks in his book "pattern recognition and machine learning".

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.