# clarification on back-propagation calculations for a fully connected neural network

I am currently taking Andrew Ng's Deep Learning Course on coursera and I couldn't get my head around how actually back-propagation in calculated.

Let's say my fully connected neural network looks like this: Notation I will be using:
X = Matrix of inputs with each row as a single example,
Y = output matrix,
L = Total Number of layers = 3,
W = weight matrix of a layer. eg: $$W^{[2]}$$ is weight matrix of layer 2,
b = bias of a layer. eg: $$b^{[2]}$$ is bias of layer 2,
Z = Linear function of a layer. eg: $$Z^{[2]}$$ is linear output of layer 2,
A = Post-activation output of a layer. $$A^{[2]}$$ is Activation of layer 2,
$$^{T}$$ = transpose of a matrix. eg: if $$A$$ is a matrix, $$A^{T}$$ is transpose of this matrix, and Loss = Loss after a Gradient Descent Iteration,
sigma = mathematical sigma used for summation,
relu = relu activation function,
$$\sigma$$= sigmoid activation function,
. = matrix multiplication and * = element-wise multiplication of a matrix.

So, during Forward Propagation, our calculations will be:

at first layer:
$$Z^{[1]} = W^{[1]} . X + b^{[1]}$$
$$A^{[1]} = relu(Z^{[1]})$$

at second layer:
$$Z^{[2]} = W^{[2]} . A^{[1]} + b^{[2]}$$
$$A^{[2]} = relu(Z^{[2]})$$

at third and output layer:
$$Z^{[3]} = W^{[3]} . A^{[2]} + b^{[3]}$$
$$A^{[3]} = \sigma(Z^{[3]})$$

Now the back-propagation (this is where my confusion starts and I may have got these equations wrong, so, correct me if I am wrong):

at third and output layer:
EDIT STARTS:
insetead of this: $$\frac{\partial A}{\partial L} = -(\frac{Y}{A^{[3]}} - \frac{1-Y}{A^{[3]}})$$
this should be done:
$$\frac{\partial A}{\partial L} = \hat{Y} - Y$$, where $$\hat{Y}$$ is output Y and $$Y$$ is true Y.
Or some form of cost measure should be used.
EDIT ENDS.
let's call $$\frac{\partial A}{\partial L}$$, $$\partial AL$$

then, $$\partial Z^{[3]} = \sigma(\partial AL)$$
$$\partial W^{[3]} = 1/m * (\partial Z^{[3]} . \partial AL^{T})$$
$$\partial b^{[3]} = 1/m * \sum(\partial Z^{[3]})$$
$$\partial A^{[2]} = W^{[3]T} . \partial Z^{[3]})$$

at second layer:

$$\partial Z^{[2]} = relu(\partial A^{[2]})$$
$$\partial W^{[2]} = 1/m * (\partial Z^{[2]} . \partial A^{[2]T})$$
$$\partial b^{[2]} = 1/m * \sum(\partial Z^{[2]})$$
$$\partial A^{[1]} = 1/m * (\partial Z^{[2]} . \partial A^{[2]T})$$

at first layer:

$$\partial Z^{[1]} = relu(\partial A^{[1]})$$
$$\partial W^{[1]} = 1/m * (\partial Z^{[1]} . \partial A^{[1]T})$$
$$\partial b^{[1]} = 1/m * \sum(\partial Z^{[1]})$$
$$\partial A^{[0]} = 1/m * (\partial Z^{[1]} . \partial A^{[1]T})$$

And now we use dW and db at a respective layer to update weights and bias at that layer. That completes a Gradient Descent iteration. Where am I wrong and what have I missed? It would be really helpful if you shed some light and help me understand calculations that take place in each iteration of back-propagation.

This is more of a clarification or doubt than a question. Please do not downvote this. I am a beginner trying to grasp concepts of neural networks.

• Markdown supports latex, it would be much easier to read – doubllle Jun 17 at 7:21
• @doubllle, I changed the formatting. – Naveen Kumar Jun 17 at 7:54
• One obvious thing I would like to point out for you is that you backpropagate your error or loss function if you want to update the weights, not the model output – doubllle Jun 17 at 8:55
• @doubllle, I am really new to this. Will you be able to write an answer correcting the equations I got incorrect or completely wrong. – Naveen Kumar Jun 17 at 9:20
• @doubllle, please check the edit in my question. I tried to make the change you suggested. – Naveen Kumar Jun 17 at 10:10

In the figure, the net has scalar input $$x$$ and output $$z$$, with scalar weights $$w_1$$ and $$w_2$$. Intermediate computing steps are explicitly shown, where $$\otimes$$ denotes the multiplication with $$p_1, p_2$$ as the multiplication products, and $$f_1, f_2$$ are activation functions. The squared error function is taken as the loss function $$\mathcal E(\mathbf w|x, d)=\frac{1}{2}(z-d)^2$$ with $$d$$ the target values.
Start with calculating the first order derivatives of $$\mathcal E(\mathbf w)$$ with respect to $$w_1$$ and $$w_2$$ \begin{align} \nonumber \frac{\partial \mathcal E}{\partial w_2} &=\frac{\partial \mathcal E}{\partial z}\frac{\partial z}{\partial p_2}\frac{\partial p_2}{\partial w_2}\\ \nonumber &=(d-z)\frac{\partial z}{\partial p_2} y \end{align} \begin{align} \nonumber \frac{\partial \mathcal E}{\partial w_1} &=\frac{\partial \mathcal E}{\partial z}\frac{\partial z}{\partial p_2}\frac{\partial p_2}{\partial y} \frac{\partial y}{\partial p_1}\frac{\partial p_1}{\partial w_1}\\ \nonumber &=(d-z)\frac{\partial z}{\partial p_2}w\frac{\partial y}{\partial p_1} x \end{align} Then the derivatives are written in vector form as $$$$g = (\frac{\partial \mathcal E}{\partial w_1}\ \frac{\partial \mathcal E}{\partial w_2})^T.$$$$ The weight vector can be updated at step $$k$$ by $$$$\mathbf w_{k+1} = \mathbf w_k-\alpha g.$$$$ The calculation of gradients $$g$$ can be extended to higher dimensions. The chain-rule based propagation of the errors is intuitive in the scalar input-output case. For the sake of illustration, the loss function here considered only one instance. To get a more rigorous and comprehensive treatment of gradient descent, you can search for stochastic gradient descent, mini-batch gradient descent, and batch gradient descent.