Numerical check of gradient in neural network

Question

I am trying to check if my implementation of backpropogation is correct by checking the calculated gradients with the numeric gradient.

I am testing it on a very simple linear network (i.e. no sigmoid on the output activation) as shown below:

There are two input neurons and one output neuron. The weights have the values [0.34, 0.56] and the bias is 0.54

The loss function used is the squared loss i.e.

$$ \frac{1}{2}(y - output)^2 $$

With an input of [1, 0], working out the gradients w.r.t the weights (dw) and bias (db) using the chain rule (backpropogation) gives me: dw = [0.7744, 0] and db = [0.7744] which is exactly what my backprop implementation gives.

However, to calculate the numerical gradient my understanding is that I should compute $g(\theta) \approx \frac{J(\theta + \epsilon) - J(\theta - \epsilon)}{2\epsilon}$ where $J$ is the network output.

Using the example input of [1,0] and perturbing the 0.34 weight with $+/- \epsilon$ the numeric gradient will always be $1$, regardless of what I use for $\epsilon$. This is quite far away from my analytic gradient.

Am I doing something wrong?

Thanks

Answer 1

What is $J(\theta)$?

If it is the output (without activation functions) directly,

$J(\theta)=ax_1+bx_2+c-y$.

Then $\frac{\partial J}{\partial a} = x_1=1$ is correct.

If it is the squared loss,

$J(\theta)=\frac{1}{2}(ax_1+bx_2+c-y)^2$,

Then,

$\frac{\partial J}{\partial a} = (ax_1+bx_2+c-y)x_1=0.34+0.54-y$,

which might be the value 0.7744 you got using backpropogation.

Perhaps you might have used different $J(\theta)$ for backpropogation and numerical check. Hope that helps.

Answer 2

The error here is simply that you should take the derivative of your loss function,

$$ E(\theta) = \frac{1}{2}\, (\, target - output\,(\,input, \, \theta)\,)^2\,, $$

and not the one of your network output $J$. So you should calculate

$$g(\theta) \approx \frac{E(\theta + \epsilon) - E(\theta - \epsilon)}{2\epsilon}$$

instead.

Rationale: you want to minimize $E$ with respect to the parameter $\theta$ (the neural network weights). So you always go a step into the direction in which the decrease of $E$ is maximal, which is given by $-\frac{\partial E}{\partial \theta_{ij}}$ (where $\theta_{ij}$ is a network weight).

Bishop gives a short section on numerical evaluation of the derivative and the hessian in neural networks in his book "pattern recognition and machine learning".