Backpropagation gradients don't match approximated gradients I am in the process of implementing back propagation into my image classification neural net. 
I am using this cost function with a sigmoid output layer and ReLU hidden layers. 

The neural net has 3 layers of sizes hidden 4 hidden 3 and output 1. 
To start, I am only trying to implement back propagation for the weights of the output layer called W3. 
The parameters dictionary holds the values of the weights and biases.
I have used this gradient approximation method to find what the gradients should be:
def check_gradients(X,Y,parameters):

    epsilon = 0.0001

    parameters1 = copy.deepcopy(parameters)
    parameters2 = copy.deepcopy(parameters)

    parameters1["W3"][:,2] += epsilon
    parameters2["W3"][:,2] -= epsilon

    fp1, fp1cache = fp.forward_propagate(parameters1, X, 4)
    fp2, fp2cache = fp.forward_propagate(parameters2, X, 4)

    cost1 = fp.cost(Y, fp1)
    cost2 = fp.cost(Y, fp2)

    gradapprox = (cost1 - cost2) / (2. *epsilon)

    return gradapprox

When I run this method with the initial weights I get -0.000651293417531 for the third element of W3.
My backpropagation method is this:
def back_propagation(X, y, al, L,  parameters, caches):


    m = X.shape[1]

    grads = {}

    dzl = - (np.divide(y, caches["a3"]) - np.divide(1-y,1-
    caches["a3"])) 
    dzl *=  a.sigmoid_back(caches["Z3"])
    grads["W" + str(L-1)] = (dzl.dot(caches["a2"].T))


return grads

str(L-1) refers to the final layer because when I input the dimension array I include the input layer so in order to have the last layer be W3 I do W + 4 - 1.
The backpropagation method returns for the third element of W3: 0.000076791652.
caches is a dictionary containing Z and a values from the forward propagation.
Here is an image of my calculations for the backpropagation of W3:

Here is my full code implementation. Just run main.py after cloning repository.
https://github.com/SamKirkiles/not-hotdog
Why is my backpropagation method not returning the same thing as gradient approximation? Or are the two values close enough that it is a correct approximation?
Thanks
 A: You are using the central differencing method to compute an approximation of the gradient. It often leads to a poor approximation.
If you want a really precise gradient approximation, use the complex step [link] which approximates the gradients as follows:
def gradient_approx(x, f_func, eps=1e-7):
  '''
  - x is the set of parameters of your model in one long vector

  - f_func is your loss function that takes the model parameters as input

  - eps is just a small number, note that this method is not very 
  sensitive to the value of eps
  '''

  e = np.zeros(x.size)

  n_params = len(block)

  gA = np.zeros(n_params)
  for i in range(n_params):
    e[i] = 1.

    val = f_func(x + e * np.complex(0, eps))
    gA[i] = np.imag(val) / eps

    e[i] = 0

  # Return the approximated gradients
  return gA

If your true gradients do not match the approximated gradient using this method, then there is probably a bug in the code!
A: I finally found the answer. I was confused because the Coursera deeplearning.ai course abstracts backpropagation for people who don't know calculus.
This paper was very helpful when finding the solution.
Here is what I had to change:
def back_propagation(X, y, al, L,  parameters, caches):


   m = X.shape[1]

    grads = {}


    """ Another way of writing this line is:
       grads["W" + str(L-1)] = (1/m) * (caches['a3']-
       y).dot(caches['a2'].T)
    """

    dzl = (-np.divide(y, al) + np.divide(1 - y, 1 - al))
    dzl *=  a.sigmoid_back(caches["a3"])
    grads["W" + str(L-1)] = (1/m) *  (dzl.dot(caches["a2"].T))

    return grads

I was putting Z3 into sigmoid derivative instead of A3 because the form of the sigmoid derivative is h(Z3)(1-h(Z3)) rather than Z3(1-Z3). It is confusing how this is phrased in the deeplearning.ai course.
Hope this helps someone
