I am currently taking Andrew Ng's Deep Learning Course on coursera and I couldn't get my head around how actually back-propagation in calculated.

Let's say my fully connected neural network looks like this: enter image description here Notation I will be using:
X = Matrix of inputs with each row as a single example,
Y = output matrix,
L = Total Number of layers = 3,
W = weight matrix of a layer. eg: $W^{[2]}$ is weight matrix of layer 2,
b = bias of a layer. eg: $b^{[2]}$ is bias of layer 2,
Z = Linear function of a layer. eg: $Z^{[2]}$ is linear output of layer 2,
A = Post-activation output of a layer. $A^{[2]}$ is Activation of layer 2,
$^{T}$ = transpose of a matrix. eg: if $A$ is a matrix, $A^{T}$ is transpose of this matrix, and Loss = Loss after a Gradient Descent Iteration,
sigma = mathematical sigma used for summation,
relu = relu activation function,
$\sigma$= sigmoid activation function,
. = matrix multiplication and * = element-wise multiplication of a matrix.

So, during Forward Propagation, our calculations will be:

at first layer:
$Z^{[1]} = W^{[1]} . X + b^{[1]}$
$A^{[1]} = relu(Z^{[1]})$

at second layer:
$Z^{[2]} = W^{[2]} . A^{[1]} + b^{[2]}$
$A^{[2]} = relu(Z^{[2]})$

at third and output layer:
$Z^{[3]} = W^{[3]} . A^{[2]} + b^{[3]}$
$A^{[3]} = \sigma(Z^{[3]})$

Now the back-propagation (this is where my confusion starts and I may have got these equations wrong, so, correct me if I am wrong):

at third and output layer:
insetead of this: $\frac{\partial A}{\partial L} = -(\frac{Y}{A^{[3]}} - \frac{1-Y}{A^{[3]}})$
this should be done:
$\frac{\partial A}{\partial L} = \hat{Y} - Y$, where $\hat{Y}$ is output Y and $Y$ is true Y.
Or some form of cost measure should be used.
let's call $\frac{\partial A}{\partial L}$, $\partial AL$

then, $\partial Z^{[3]} = \sigma(\partial AL)$
$\partial W^{[3]} = 1/m * (\partial Z^{[3]} . \partial AL^{T})$
$\partial b^{[3]} = 1/m * \sum(\partial Z^{[3]})$
$\partial A^{[2]} = W^{[3]T} . \partial Z^{[3]})$

at second layer:

$\partial Z^{[2]} = relu(\partial A^{[2]})$
$\partial W^{[2]} = 1/m * (\partial Z^{[2]} . \partial A^{[2]T})$
$\partial b^{[2]} = 1/m * \sum(\partial Z^{[2]})$
$\partial A^{[1]} = 1/m * (\partial Z^{[2]} . \partial A^{[2]T})$

at first layer:

$\partial Z^{[1]} = relu(\partial A^{[1]})$
$\partial W^{[1]} = 1/m * (\partial Z^{[1]} . \partial A^{[1]T})$
$\partial b^{[1]} = 1/m * \sum(\partial Z^{[1]})$
$\partial A^{[0]} = 1/m * (\partial Z^{[1]} . \partial A^{[1]T})$

And now we use dW and db at a respective layer to update weights and bias at that layer. That completes a Gradient Descent iteration. Where am I wrong and what have I missed? It would be really helpful if you shed some light and help me understand calculations that take place in each iteration of back-propagation.

This is more of a clarification or doubt than a question. Please do not downvote this. I am a beginner trying to grasp concepts of neural networks.

  • 1
    $\begingroup$ Markdown supports latex, it would be much easier to read $\endgroup$ – doubllle Jun 17 at 7:21
  • $\begingroup$ @doubllle, I changed the formatting. $\endgroup$ – Naveen Kumar Jun 17 at 7:54
  • 1
    $\begingroup$ One obvious thing I would like to point out for you is that you backpropagate your error or loss function if you want to update the weights, not the model output $\endgroup$ – doubllle Jun 17 at 8:55
  • $\begingroup$ @doubllle, I am really new to this. Will you be able to write an answer correcting the equations I got incorrect or completely wrong. $\endgroup$ – Naveen Kumar Jun 17 at 9:20
  • $\begingroup$ @doubllle, please check the edit in my question. I tried to make the change you suggested. $\endgroup$ – Naveen Kumar Jun 17 at 10:10

If I am allowed to say so, your formulation actually makes things a bit more complicated, given that some basic concepts are not clear to you. For a comprehensive intro of BP, you can have a look at this.

I'll just give you a scalar case for illustrating the basic ideas. Hope it helps.

scalar neural net In the figure, the net has scalar input $x$ and output $z$, with scalar weights $w_1$ and $w_2$. Intermediate computing steps are explicitly shown, where $\otimes$ denotes the multiplication with $p_1, p_2$ as the multiplication products, and $f_1, f_2$ are activation functions. The squared error function is taken as the loss function $\mathcal E(\mathbf w|x, d)=\frac{1}{2}(z-d)^2$ with $d$ the target values.
Start with calculating the first order derivatives of $\mathcal E(\mathbf w)$ with respect to $w_1$ and $w_2$ \begin{align} \nonumber \frac{\partial \mathcal E}{\partial w_2} &=\frac{\partial \mathcal E}{\partial z}\frac{\partial z}{\partial p_2}\frac{\partial p_2}{\partial w_2}\\ \nonumber &=(d-z)\frac{\partial z}{\partial p_2} y \end{align} \begin{align} \nonumber \frac{\partial \mathcal E}{\partial w_1} &=\frac{\partial \mathcal E}{\partial z}\frac{\partial z}{\partial p_2}\frac{\partial p_2}{\partial y} \frac{\partial y}{\partial p_1}\frac{\partial p_1}{\partial w_1}\\ \nonumber &=(d-z)\frac{\partial z}{\partial p_2}w\frac{\partial y}{\partial p_1} x \end{align} Then the derivatives are written in vector form as \begin{equation} g = (\frac{\partial \mathcal E}{\partial w_1}\ \frac{\partial \mathcal E}{\partial w_2})^T. \end{equation} The weight vector can be updated at step $k$ by \begin{equation} \mathbf w_{k+1} = \mathbf w_k-\alpha g. \end{equation} The calculation of gradients $g$ can be extended to higher dimensions. The chain-rule based propagation of the errors is intuitive in the scalar input-output case. For the sake of illustration, the loss function here considered only one instance. To get a more rigorous and comprehensive treatment of gradient descent, you can search for stochastic gradient descent, mini-batch gradient descent, and batch gradient descent.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.