clarification on back-propagation calculations for a fully connected neural network

Question

I am currently taking Andrew Ng's Deep Learning Course on coursera and I couldn't get my head around how actually back-propagation in calculated.

Let's say my fully connected neural network looks like this: Notation I will be using:
X = Matrix of inputs with each row as a single example,
Y = output matrix,
L = Total Number of layers = 3,
W = weight matrix of a layer. eg: $W^{[2]}$ is weight matrix of layer 2,
b = bias of a layer. eg: $b^{[2]}$ is bias of layer 2,
Z = Linear function of a layer. eg: $Z^{[2]}$ is linear output of layer 2,
A = Post-activation output of a layer. $A^{[2]}$ is Activation of layer 2,
$^{T}$ = transpose of a matrix. eg: if $A$ is a matrix, $A^{T}$ is transpose of this matrix, and Loss = Loss after a Gradient Descent Iteration,
sigma = mathematical sigma used for summation,
relu = relu activation function,
$\sigma$= sigmoid activation function,
. = matrix multiplication and * = element-wise multiplication of a matrix.

So, during Forward Propagation, calculations will be:

at first layer:
$Z^{[1]} = W^{[1]} . X + b^{[1]}$
$A^{[1]} = relu(Z^{[1]})$

at second layer:
$Z^{[2]} = W^{[2]} . A^{[1]} + b^{[2]}$
$A^{[2]} = relu(Z^{[2]})$

at third and output layer:
$Z^{[3]} = W^{[3]} . A^{[2]} + b^{[3]}$
$A^{[3]} = \sigma(Z^{[3]})$

Now the back-propagation (this is where my confusion starts and I may have got these equations wrong, so, correct me if I am wrong):

at third and output layer:

$\frac{\partial A}{\partial L} = -(\frac{Y}{A^{[3]}} - \frac{1-Y}{A^{[3]}})$
this should be done:
$\frac{\partial A}{\partial L} = \hat{Y} - Y$, where $\hat{Y}$ is output Y and $Y$ is true Y.
Or some form of cost measure should be used.

let's call $\frac{\partial A}{\partial L}$, $\partial AL$

then, $\partial Z^{[3]} = \sigma(\partial AL)$
$\partial W^{[3]} = 1/m * (\partial Z^{[3]} . \partial AL^{T})$
$\partial b^{[3]} = 1/m * \sum(\partial Z^{[3]})$
$\partial A^{[2]} = W^{[3]T} . \partial Z^{[3]})$

at second layer:

$\partial Z^{[2]} = relu(\partial A^{[2]})$
$\partial W^{[2]} = 1/m * (\partial Z^{[2]} . \partial A^{[2]T})$
$\partial b^{[2]} = 1/m * \sum(\partial Z^{[2]})$
$\partial A^{[1]} = 1/m * (\partial Z^{[2]} . \partial A^{[2]T})$

at first layer:

$\partial Z^{[1]} = relu(\partial A^{[1]})$
$\partial W^{[1]} = 1/m * (\partial Z^{[1]} . \partial A^{[1]T})$
$\partial b^{[1]} = 1/m * \sum(\partial Z^{[1]})$
$\partial A^{[0]} = 1/m * (\partial Z^{[1]} . \partial A^{[1]T})$

then based on the optimiser, weights will be updated. and then the forward-pass, loss-calcualtion, backward-pass, updating weights and son on.

And now we use $\partial W$ and $\partial b$ at a respective layer to update weights and bias at that layer. That completes a Gradient Descent iteration. Where am I wrong and what have I missed? It would be really helpful if you shed some light and help me understand calculations that take place in each iteration of back-propagation.

Answer 1

If I am allowed to say so, your formulation actually makes things a bit more complicated, given that some basic concepts are not clear to you. For a comprehensive intro of BP, you can have a look at this.

I'll just give you a scalar case for illustrating the basic ideas. Hope it helps.

In the figure, the net has scalar input $x$ and output $z$, with scalar weights $w_1$ and $w_2$. Intermediate computing steps are explicitly shown, where $\otimes$ denotes the multiplication with $p_1, p_2$ as the multiplication products, and $f_1, f_2$ are activation functions. The squared error function is taken as the loss function $\mathcal E(\mathbf w|x, d)=\frac{1}{2}(z-d)^2$ with $d$ the target values.
Start with calculating the first order derivatives of $\mathcal E(\mathbf w)$ with respect to $w_1$ and $w_2$ \begin{align} \nonumber \frac{\partial \mathcal E}{\partial w_2} &=\frac{\partial \mathcal E}{\partial z}\frac{\partial z}{\partial p_2}\frac{\partial p_2}{\partial w_2}\\ \nonumber &=(d-z)\frac{\partial z}{\partial p_2} y \end{align} \begin{align} \nonumber \frac{\partial \mathcal E}{\partial w_1} &=\frac{\partial \mathcal E}{\partial z}\frac{\partial z}{\partial p_2}\frac{\partial p_2}{\partial y} \frac{\partial y}{\partial p_1}\frac{\partial p_1}{\partial w_1}\\ \nonumber &=(d-z)\frac{\partial z}{\partial p_2}w\frac{\partial y}{\partial p_1} x \end{align} Then the derivatives are written in vector form as \begin{equation} g = (\frac{\partial \mathcal E}{\partial w_1}\ \frac{\partial \mathcal E}{\partial w_2})^T. \end{equation} The weight vector can be updated at step $k$ by \begin{equation} \mathbf w_{k+1} = \mathbf w_k-\alpha g. \end{equation} The calculation of gradients $g$ can be extended to higher dimensions. The chain-rule based propagation of the errors is intuitive in the scalar input-output case. For the sake of illustration, the loss function here considered only one instance. To get a more rigorous and comprehensive treatment of gradient descent, you can search for stochastic gradient descent, mini-batch gradient descent, and batch gradient descent.