I am currently taking Andrew Ng's Deep Learning Course on coursera and I couldn't get my head around how actually back-propagation in calculated.

Let's say my fully connected neural network looks like this: enter image description here Notation I will be using:
X = Matrix of inputs with each row as a single example,
Y = output matrix,
L = Total Number of layers = 3,
W = weight matrix of a layer. eg: $W^{[2]}$ is weight matrix of layer 2,
b = bias of a layer. eg: $b^{[2]}$ is bias of layer 2,
Z = Linear function of a layer. eg: $Z^{[2]}$ is linear output of layer 2,
A = Post-activation output of a layer. $A^{[2]}$ is Activation of layer 2,
$^{T}$ = transpose of a matrix. eg: if $A$ is a matrix, $A^{T}$ is transpose of this matrix, and Loss = Loss after a Gradient Descent Iteration,
sigma = mathematical sigma used for summation,
relu = relu activation function,
$\sigma$= sigmoid activation function,
. = matrix multiplication and * = element-wise multiplication of a matrix.

So, during Forward Propagation, calculations will be:

at first layer:
$Z^{[1]} = W^{[1]} . X + b^{[1]}$
$A^{[1]} = relu(Z^{[1]})$

at second layer:
$Z^{[2]} = W^{[2]} . A^{[1]} + b^{[2]}$
$A^{[2]} = relu(Z^{[2]})$

at third and output layer:
$Z^{[3]} = W^{[3]} . A^{[2]} + b^{[3]}$
$A^{[3]} = \sigma(Z^{[3]})$

Now the back-propagation (this is where my confusion starts and I may have got these equations wrong, so, correct me if I am wrong):

at third and output layer:

$\frac{\partial A}{\partial L} = -(\frac{Y}{A^{[3]}} - \frac{1-Y}{A^{[3]}})$
this should be done:
$\frac{\partial A}{\partial L} = \hat{Y} - Y$, where $\hat{Y}$ is output Y and $Y$ is true Y.
Or some form of cost measure should be used.

let's call $\frac{\partial A}{\partial L}$, $\partial AL$

then, $\partial Z^{[3]} = \sigma(\partial AL)$
$\partial W^{[3]} = 1/m * (\partial Z^{[3]} . \partial AL^{T})$
$\partial b^{[3]} = 1/m * \sum(\partial Z^{[3]})$
$\partial A^{[2]} = W^{[3]T} . \partial Z^{[3]})$

at second layer:

$\partial Z^{[2]} = relu(\partial A^{[2]})$
$\partial W^{[2]} = 1/m * (\partial Z^{[2]} . \partial A^{[2]T})$
$\partial b^{[2]} = 1/m * \sum(\partial Z^{[2]})$
$\partial A^{[1]} = 1/m * (\partial Z^{[2]} . \partial A^{[2]T})$

at first layer:

$\partial Z^{[1]} = relu(\partial A^{[1]})$
$\partial W^{[1]} = 1/m * (\partial Z^{[1]} . \partial A^{[1]T})$
$\partial b^{[1]} = 1/m * \sum(\partial Z^{[1]})$
$\partial A^{[0]} = 1/m * (\partial Z^{[1]} . \partial A^{[1]T})$

then based on the optimiser, weights will be updated. and then the forward-pass, loss-calcualtion, backward-pass, updating weights and son on.

And now we use $\partial W$ and $\partial b$ at a respective layer to update weights and bias at that layer. That completes a Gradient Descent iteration. Where am I wrong and what have I missed? It would be really helpful if you shed some light and help me understand calculations that take place in each iteration of back-propagation.

  • 1
    $\begingroup$ Markdown supports latex, it would be much easier to read $\endgroup$
    – doubllle
    Jun 17, 2020 at 7:21
  • $\begingroup$ @doubllle, I changed the formatting. $\endgroup$ Jun 17, 2020 at 7:54
  • 1
    $\begingroup$ One obvious thing I would like to point out for you is that you backpropagate your error or loss function if you want to update the weights, not the model output $\endgroup$
    – doubllle
    Jun 17, 2020 at 8:55
  • $\begingroup$ @doubllle, I am really new to this. Will you be able to write an answer correcting the equations I got incorrect or completely wrong. $\endgroup$ Jun 17, 2020 at 9:20
  • $\begingroup$ @doubllle, please check the edit in my question. I tried to make the change you suggested. $\endgroup$ Jun 17, 2020 at 10:10

1 Answer 1


If I am allowed to say so, your formulation actually makes things a bit more complicated, given that some basic concepts are not clear to you. For a comprehensive intro of BP, you can have a look at this.

I'll just give you a scalar case for illustrating the basic ideas. Hope it helps.

scalar neural net In the figure, the net has scalar input $x$ and output $z$, with scalar weights $w_1$ and $w_2$. Intermediate computing steps are explicitly shown, where $\otimes$ denotes the multiplication with $p_1, p_2$ as the multiplication products, and $f_1, f_2$ are activation functions. The squared error function is taken as the loss function $\mathcal E(\mathbf w|x, d)=\frac{1}{2}(z-d)^2$ with $d$ the target values.
Start with calculating the first order derivatives of $\mathcal E(\mathbf w)$ with respect to $w_1$ and $w_2$ \begin{align} \nonumber \frac{\partial \mathcal E}{\partial w_2} &=\frac{\partial \mathcal E}{\partial z}\frac{\partial z}{\partial p_2}\frac{\partial p_2}{\partial w_2}\\ \nonumber &=(d-z)\frac{\partial z}{\partial p_2} y \end{align} \begin{align} \nonumber \frac{\partial \mathcal E}{\partial w_1} &=\frac{\partial \mathcal E}{\partial z}\frac{\partial z}{\partial p_2}\frac{\partial p_2}{\partial y} \frac{\partial y}{\partial p_1}\frac{\partial p_1}{\partial w_1}\\ \nonumber &=(d-z)\frac{\partial z}{\partial p_2}w\frac{\partial y}{\partial p_1} x \end{align} Then the derivatives are written in vector form as \begin{equation} g = (\frac{\partial \mathcal E}{\partial w_1}\ \frac{\partial \mathcal E}{\partial w_2})^T. \end{equation} The weight vector can be updated at step $k$ by \begin{equation} \mathbf w_{k+1} = \mathbf w_k-\alpha g. \end{equation} The calculation of gradients $g$ can be extended to higher dimensions. The chain-rule based propagation of the errors is intuitive in the scalar input-output case. For the sake of illustration, the loss function here considered only one instance. To get a more rigorous and comprehensive treatment of gradient descent, you can search for stochastic gradient descent, mini-batch gradient descent, and batch gradient descent.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.