# clarification on back-propagation calculations for a fully connected neural network

I am currently taking Andrew Ng's Deep Learning Course on coursera and I couldn't get my head around how actually back-propagation in calculated.

Let's say my fully connected neural network looks like this: Notation I will be using:
X = Matrix of inputs with each row as a single example,
Y = output matrix,
L = Total Number of layers = 3,
W = weight matrix of a layer. eg: $$W^{}$$ is weight matrix of layer 2,
b = bias of a layer. eg: $$b^{}$$ is bias of layer 2,
Z = Linear function of a layer. eg: $$Z^{}$$ is linear output of layer 2,
A = Post-activation output of a layer. $$A^{}$$ is Activation of layer 2,
$$^{T}$$ = transpose of a matrix. eg: if $$A$$ is a matrix, $$A^{T}$$ is transpose of this matrix, and Loss = Loss after a Gradient Descent Iteration,
sigma = mathematical sigma used for summation,
relu = relu activation function,
$$\sigma$$= sigmoid activation function,
. = matrix multiplication and * = element-wise multiplication of a matrix.

So, during Forward Propagation, our calculations will be:

at first layer:
$$Z^{} = W^{} . X + b^{}$$
$$A^{} = relu(Z^{})$$

at second layer:
$$Z^{} = W^{} . A^{} + b^{}$$
$$A^{} = relu(Z^{})$$

at third and output layer:
$$Z^{} = W^{} . A^{} + b^{}$$
$$A^{} = \sigma(Z^{})$$

Now the back-propagation (this is where my confusion starts and I may have got these equations wrong, so, correct me if I am wrong):

at third and output layer:
EDIT STARTS:
insetead of this: $$\frac{\partial A}{\partial L} = -(\frac{Y}{A^{}} - \frac{1-Y}{A^{}})$$
this should be done:
$$\frac{\partial A}{\partial L} = \hat{Y} - Y$$, where $$\hat{Y}$$ is output Y and $$Y$$ is true Y.
Or some form of cost measure should be used.
EDIT ENDS.
let's call $$\frac{\partial A}{\partial L}$$, $$\partial AL$$

then, $$\partial Z^{} = \sigma(\partial AL)$$
$$\partial W^{} = 1/m * (\partial Z^{} . \partial AL^{T})$$
$$\partial b^{} = 1/m * \sum(\partial Z^{})$$
$$\partial A^{} = W^{T} . \partial Z^{})$$

at second layer:

$$\partial Z^{} = relu(\partial A^{})$$
$$\partial W^{} = 1/m * (\partial Z^{} . \partial A^{T})$$
$$\partial b^{} = 1/m * \sum(\partial Z^{})$$
$$\partial A^{} = 1/m * (\partial Z^{} . \partial A^{T})$$

at first layer:

$$\partial Z^{} = relu(\partial A^{})$$
$$\partial W^{} = 1/m * (\partial Z^{} . \partial A^{T})$$
$$\partial b^{} = 1/m * \sum(\partial Z^{})$$
$$\partial A^{} = 1/m * (\partial Z^{} . \partial A^{T})$$

And now we use dW and db at a respective layer to update weights and bias at that layer. That completes a Gradient Descent iteration. Where am I wrong and what have I missed? It would be really helpful if you shed some light and help me understand calculations that take place in each iteration of back-propagation.

This is more of a clarification or doubt than a question. Please do not downvote this. I am a beginner trying to grasp concepts of neural networks.

• Markdown supports latex, it would be much easier to read – doubllle Jun 17 at 7:21
• @doubllle, I changed the formatting. – Naveen Kumar Jun 17 at 7:54
• One obvious thing I would like to point out for you is that you backpropagate your error or loss function if you want to update the weights, not the model output – doubllle Jun 17 at 8:55
• @doubllle, I am really new to this. Will you be able to write an answer correcting the equations I got incorrect or completely wrong. – Naveen Kumar Jun 17 at 9:20
• @doubllle, please check the edit in my question. I tried to make the change you suggested. – Naveen Kumar Jun 17 at 10:10 In the figure, the net has scalar input $$x$$ and output $$z$$, with scalar weights $$w_1$$ and $$w_2$$. Intermediate computing steps are explicitly shown, where $$\otimes$$ denotes the multiplication with $$p_1, p_2$$ as the multiplication products, and $$f_1, f_2$$ are activation functions. The squared error function is taken as the loss function $$\mathcal E(\mathbf w|x, d)=\frac{1}{2}(z-d)^2$$ with $$d$$ the target values.
Start with calculating the first order derivatives of $$\mathcal E(\mathbf w)$$ with respect to $$w_1$$ and $$w_2$$ \begin{align} \nonumber \frac{\partial \mathcal E}{\partial w_2} &=\frac{\partial \mathcal E}{\partial z}\frac{\partial z}{\partial p_2}\frac{\partial p_2}{\partial w_2}\\ \nonumber &=(d-z)\frac{\partial z}{\partial p_2} y \end{align} \begin{align} \nonumber \frac{\partial \mathcal E}{\partial w_1} &=\frac{\partial \mathcal E}{\partial z}\frac{\partial z}{\partial p_2}\frac{\partial p_2}{\partial y} \frac{\partial y}{\partial p_1}\frac{\partial p_1}{\partial w_1}\\ \nonumber &=(d-z)\frac{\partial z}{\partial p_2}w\frac{\partial y}{\partial p_1} x \end{align} Then the derivatives are written in vector form as $$\begin{equation} g = (\frac{\partial \mathcal E}{\partial w_1}\ \frac{\partial \mathcal E}{\partial w_2})^T. \end{equation}$$ The weight vector can be updated at step $$k$$ by $$\begin{equation} \mathbf w_{k+1} = \mathbf w_k-\alpha g. \end{equation}$$ The calculation of gradients $$g$$ can be extended to higher dimensions. The chain-rule based propagation of the errors is intuitive in the scalar input-output case. For the sake of illustration, the loss function here considered only one instance. To get a more rigorous and comprehensive treatment of gradient descent, you can search for stochastic gradient descent, mini-batch gradient descent, and batch gradient descent.