I've been taking Andrew Ng's course on Coursera, and although it has been great so far, I loathe his lack of supplementary documents on proofs. Thankfully, there are some great articles found pretty good article on the backpropagation proof, namely, this one by Sudeep Raja.

There are some notation differences and even some formula differences that are really throwing me off and would like more insight. Here is the slide specific to Andrew's lecture.

For notation's sake, assume there are three layers, with $\Theta^1, \Theta^2, \Theta^3$ as the weights. $a^1=x$ is the input, $a^2$ is the output of layer 1, and $a^4$ is the expected output of layer 3 and the final output.

$\Delta^1$ is $\partial J/\partial\Theta^1$, $\Delta^2$ is $\partial J/\partial\Theta^2$, ...

Note: This notation is identical to Andrew's but different from Sudeep's.

Andrew has:

  • $\delta^{(4)} = a^4-\vec y$,
  • $\delta^3=(\Theta^3)^T\delta^4.*g\prime(z^3)$,
  • $\Delta^3=\delta^4(a^4)^T$

Sudeep has

  • $\delta^{(4)} = (a^4-\vec y).*g\prime(z^4)$,
  • $\delta^3=(\Theta^3)^T\delta^4.*g\prime(z^3)$,
  • $\Delta^3=\delta^4(a^4)^T$

Notice the difference in $\delta^4$. Why?? Update: Answered in edits.

Also, in Sudeep's proof, when he's deriving $\partial E/\partial W_2$, he got $$ \frac{\partial E}{\partial W_2} = \delta_3 \frac{\partial}{\partial W_2}(W_3x_2) = W_3^T\delta_3\frac{\partial}{\partial W_2}x_2 $$

$W_3$ is a constant so it comes out, but why is it transposed? I understand that you have to to get the dimensions to line up, but I don't think that's a sufficient argument as to why you do it.

Lastly, in Sudeep's proof, when he's deriving $\partial E/\partial W_3$, he got $$ \frac{\partial}{\partial W_3}(W_3x_2) = x_2^T $$ Again, why is $x_2$ transposed?


After more googling, I've learned that the reason Andrew's solution doesn't have the $g(z^4)$ is because he's using the cross-entropy cost function, explained further here, which leads to $g\prime(z)$ being canceled.

  • 2
    $\begingroup$ The book: Neural Network Design ( Hagan, Demuth..) provided a nice and not-complicated solution for the backpropagation algorithm, my lecture suggested me that book :)) Hope you understand thoughtfully! $\endgroup$
    – Duong Hang
    Commented Feb 10, 2019 at 15:39
  • $\begingroup$ Here's an article giving a vectorised proof of Andrew Ng's formulas. towardsdatascience.com/… The plus here is that not so many indices are used, and you can clearly see where the transpose and the order of matrix multiplication come from. $\endgroup$
    – btt
    Commented Dec 1, 2019 at 14:09

1 Answer 1


Here, the derivatives are taken with respect to the vector, as a result, rules of differentiation are slightly different. My suggestion is when in doubt, perform derivation for each index of the vector and then extend to the general case. For example, to answer the second question you can find derivatives for each index, e.g. $\frac{\partial E} {\partial W^{j}_2}$ and then generalize. Similarly for $\frac{\partial E} {\partial W^{j}_3}$

  • $\begingroup$ Is it respect to a vector or a matrix? Because I'm trying to do it with two indexes $i,j$ and failing miserably. I understand that the product $W_3x_2$ is a vector, but $\partial / \partial W_2$ is a matrix differentiation, so shouldn't you do matrix differentiation? E.g., $\frac{\partial}{\partial (W_2)_{11}} (W_3x_2) = (W_3)_{11} \partial (x_2)_{11}$, $\frac{\partial}{\partial (W_2)_{12}} = (W_3)_{12} \partial (x_2)_{12}$, .. in which case I'm getting the matrix $W_3$, not $W_3^T$ $\endgroup$
    – azizj
    Commented Aug 7, 2017 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.