Assume this neural net.


In our course we had to provide the forward and backward pass for this net. This is the solution we received:


Even with the solutions provided I still have problems understanding the steps taken to get through the backward pass. Especially the step $\frac{\partial L}{\partial z_0}$ where the partial derivative of the skip node comes together is a mystery to me. Our Profs do not go in detail of the algorithm. Especially if the partial is shortened it yields a contradiction, right? Or does this simply not matter since the gradient is a vector?

\begin{align} \frac{\partial L}{\partial z_0}&= \frac{\partial L}{\partial a_1}\cdot\frac{\partial a_1}{\partial z_0}+ \frac{\partial L}{\partial a_2}\cdot\frac{\partial a_2}{\partial z_0}\\ \frac{\partial L}{\partial z_0}&= \frac{\partial L}{\partial z_0}+\frac{\partial L}{\partial z_0}\\ \frac{\partial L}{\partial z_0}&\stackrel{!}{=}2\cdot\frac{\partial L}{\partial z_0}\\ \end{align}

Wheres my fallacy?

  • $\begingroup$ Hi, welcome. Where are these pictures from? $\endgroup$ – Jim Feb 8 at 19:08
  • $\begingroup$ Thank you. From our exercises. Is this not allowed? $\endgroup$ – ManuelSchneid3r Feb 8 at 19:55
  • $\begingroup$ It is allowed, but could you post a link. $\endgroup$ – Jim Feb 8 at 20:10
  • $\begingroup$ The exercises are on a closed platform of our university. $\endgroup$ – ManuelSchneid3r Feb 11 at 11:13
  • $\begingroup$ Why the downvotes? $\endgroup$ – ManuelSchneid3r Feb 11 at 11:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.