0
$\begingroup$

I am following this guide on calculating the partial derivatives of weights and biases:

https://www.datahubbs.com/deep-learning-101-the-theory/

Here it is using 1 hidden layer. How can I calculate the backpropagation if I add another hidden layer? Assuming it is using the sigmoid activation function same as the guide. Thanks!

$\endgroup$
0
$\begingroup$

The short answer: you simply continue using the chain rule and sequentially multiply your partial derivatives. I see that the backprop notes you linked use the chain rule to compute those partials, so this can easily be generalized to arbitrary neural nets.

You can start with this observation: let h_1 and h_2 be hidden layers 1 and 2 in a neural net. Let y be the output.

Originally, you had only one layer, so you simply found (dy/dh_1) and was done.

Currently, you should be able to find (dy/dh_2) simply by following your notes. So how do we compute (dy/dh_1)? Note by the chain rule, (dy/dh_1) = (dy/dh_2) * (dh_2/dh_1). We calculated one of those. So what is (dh_2/dh_1)? Write out the forward-prop equations and find out!

Sadly it's extremely difficult to type out mathematical equations here, so I've linked some backpropagation derivation notes that I found useful.

$\endgroup$
  • $\begingroup$ Thank you! The link was helpful too. $\endgroup$ – NewGirl Jan 6 at 21:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.