I am in the process of implementing back propagation into my image classification neural net.

I am using this cost function with a sigmoid output layer and ReLU hidden layers.

The neural net has 3 layers of sizes hidden 4 hidden 3 and output 1.

To start, I am only trying to implement back propagation for the weights of the output layer called W3.

The parameters dictionary holds the values of the weights and biases.

I have used this gradient approximation method to find what the gradients should be:

def check_gradients(X,Y,parameters):

epsilon = 0.0001

parameters1 = copy.deepcopy(parameters)
parameters2 = copy.deepcopy(parameters)

parameters1["W3"][:,2] += epsilon
parameters2["W3"][:,2] -= epsilon

fp1, fp1cache = fp.forward_propagate(parameters1, X, 4)
fp2, fp2cache = fp.forward_propagate(parameters2, X, 4)

cost1 = fp.cost(Y, fp1)
cost2 = fp.cost(Y, fp2)

gradapprox = (cost1 - cost2) / (2. *epsilon)



When I run this method with the initial weights I get -0.000651293417531 for the third element of W3.

My backpropagation method is this:

def back_propagation(X, y, al, L,  parameters, caches):

m = X.shape[1]

dzl = - (np.divide(y, caches["a3"]) - np.divide(1-y,1-
caches["a3"]))
dzl *=  a.sigmoid_back(caches["Z3"])



str(L-1) refers to the final layer because when I input the dimension array I include the input layer so in order to have the last layer be W3 I do W + 4 - 1.

The backpropagation method returns for the third element of W3: 0.000076791652.

caches is a dictionary containing Z and a values from the forward propagation.

Here is an image of my calculations for the backpropagation of W3:

Here is my full code implementation. Just run main.py after cloning repository.

https://github.com/SamKirkiles/not-hotdog

Why is my backpropagation method not returning the same thing as gradient approximation? Or are the two values close enough that it is a correct approximation?

Thanks

You are using the central differencing method to compute an approximation of the gradient. It often leads to a poor approximation.

If you want a really precise gradient approximation, use the complex step [link] which approximates the gradients as follows:

def gradient_approx(x, f_func, eps=1e-7):
'''
- x is the set of parameters of your model in one long vector

- f_func is your loss function that takes the model parameters as input

- eps is just a small number, note that this method is not very
sensitive to the value of eps
'''

e = np.zeros(x.size)

n_params = len(block)

gA = np.zeros(n_params)
for i in range(n_params):
e[i] = 1.

val = f_func(x + e * np.complex(0, eps))
gA[i] = np.imag(val) / eps

e[i] = 0

return gA


If your true gradients do not match the approximated gradient using this method, then there is probably a bug in the code!

• Thanks my method turned out to work fine thanks for the response though :) – s_kirkiles Jan 11 '18 at 4:43

I finally found the answer. I was confused because the Coursera deeplearning.ai course abstracts backpropagation for people who don't know calculus.

This paper was very helpful when finding the solution.

Here is what I had to change:

def back_propagation(X, y, al, L,  parameters, caches):

m = X.shape[1]

""" Another way of writing this line is:
grads["W" + str(L-1)] = (1/m) * (caches['a3']-
y).dot(caches['a2'].T)
"""

dzl = (-np.divide(y, al) + np.divide(1 - y, 1 - al))
dzl *=  a.sigmoid_back(caches["a3"])
grads["W" + str(L-1)] = (1/m) *  (dzl.dot(caches["a2"].T))