I am in the process of implementing back propagation into my image classification neural net.

I am using this cost function with a sigmoid output layer and ReLU hidden layers.

enter image description here

The neural net has 3 layers of sizes hidden 4 hidden 3 and output 1.

To start, I am only trying to implement back propagation for the weights of the output layer called W3.

The parameters dictionary holds the values of the weights and biases.

I have used this gradient approximation method to find what the gradients should be:

def check_gradients(X,Y,parameters):

    epsilon = 0.0001

    parameters1 = copy.deepcopy(parameters)
    parameters2 = copy.deepcopy(parameters)

    parameters1["W3"][:,2] += epsilon
    parameters2["W3"][:,2] -= epsilon

    fp1, fp1cache = fp.forward_propagate(parameters1, X, 4)
    fp2, fp2cache = fp.forward_propagate(parameters2, X, 4)

    cost1 = fp.cost(Y, fp1)
    cost2 = fp.cost(Y, fp2)

    gradapprox = (cost1 - cost2) / (2. *epsilon)

    return gradapprox

When I run this method with the initial weights I get -0.000651293417531 for the third element of W3.

My backpropagation method is this:

def back_propagation(X, y, al, L,  parameters, caches):

    m = X.shape[1]

    grads = {}

    dzl = - (np.divide(y, caches["a3"]) - np.divide(1-y,1-
    dzl *=  a.sigmoid_back(caches["Z3"])
    grads["W" + str(L-1)] = (dzl.dot(caches["a2"].T))

return grads

str(L-1) refers to the final layer because when I input the dimension array I include the input layer so in order to have the last layer be W3 I do W + 4 - 1.

The backpropagation method returns for the third element of W3: 0.000076791652.

caches is a dictionary containing Z and a values from the forward propagation.

Here is an image of my calculations for the backpropagation of W3:

enter image description here

Here is my full code implementation. Just run main.py after cloning repository.


Why is my backpropagation method not returning the same thing as gradient approximation? Or are the two values close enough that it is a correct approximation?



2 Answers 2


You are using the central differencing method to compute an approximation of the gradient. It often leads to a poor approximation.

If you want a really precise gradient approximation, use the complex step [link] which approximates the gradients as follows:

def gradient_approx(x, f_func, eps=1e-7):
  - x is the set of parameters of your model in one long vector

  - f_func is your loss function that takes the model parameters as input

  - eps is just a small number, note that this method is not very 
  sensitive to the value of eps

  e = np.zeros(x.size)

  n_params = len(block)

  gA = np.zeros(n_params)
  for i in range(n_params):
    e[i] = 1.

    val = f_func(x + e * np.complex(0, eps))
    gA[i] = np.imag(val) / eps

    e[i] = 0

  # Return the approximated gradients
  return gA

If your true gradients do not match the approximated gradient using this method, then there is probably a bug in the code!

  • 1
    $\begingroup$ Thanks my method turned out to work fine thanks for the response though :) $\endgroup$
    – random0620
    Commented Jan 11, 2018 at 4:43

I finally found the answer. I was confused because the Coursera deeplearning.ai course abstracts backpropagation for people who don't know calculus.

This paper was very helpful when finding the solution.

Here is what I had to change:

def back_propagation(X, y, al, L,  parameters, caches):

   m = X.shape[1]

    grads = {}

    """ Another way of writing this line is:
       grads["W" + str(L-1)] = (1/m) * (caches['a3']-

    dzl = (-np.divide(y, al) + np.divide(1 - y, 1 - al))
    dzl *=  a.sigmoid_back(caches["a3"])
    grads["W" + str(L-1)] = (1/m) *  (dzl.dot(caches["a2"].T))

    return grads

I was putting Z3 into sigmoid derivative instead of A3 because the form of the sigmoid derivative is h(Z3)(1-h(Z3)) rather than Z3(1-Z3). It is confusing how this is phrased in the deeplearning.ai course.

Hope this helps someone


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.