1
$\begingroup$

Suppose there is a DNN like this:

$h = t(w_1x+b_1)$
$g = t(w_2h+b_2)$
$\hat y = t(w_3g+b_3)$
$Loss = \frac 12\sum (y - \hat y)^2$

Just one input $x$
$t$ is activation function.

Could you explain with equations how the gradients of $w_1$, $w_2$, $w_3$ is calculated?

By the way, when combining partial derivative and chain rule,

$\frac {dk}{dg}\frac {dg}{dh}\frac {dh}{dx}$ -> $\frac {\partial k}{\partial g}\frac {\partial g}{\partial h}\frac {\partial h}{\partial w_1}$

is the above fine?

$\endgroup$
2
  • $\begingroup$ how is y related to h,g and k? $\endgroup$
    – gunes
    Dec 28, 2019 at 9:09
  • $\begingroup$ I edited my question. sorry. $\endgroup$
    – Crispy13
    Dec 28, 2019 at 9:14

1 Answer 1

1
$\begingroup$

We just execute the chain rule: $$\frac{\partial L}{\partial w_1}=\frac{\partial L}{\partial \hat y}\frac{\partial \hat y}{\partial g}\frac{\partial g}{\partial h}\frac{\partial h}{\partial w_1}$$

Then, we calculate each term separately, e.g. $$\frac{\partial h}{\partial w_1}=t'(w_1x+b_1)x,\frac{\partial g}{\partial h}=t'(w_2h+b_2)w_2,\cdots$$

Your chain rule is not fine, because we don't ever calculate $\frac{\partial h}{\partial x}$ since $x$ is not a parameter nor an outcome.

$\endgroup$
4
  • $\begingroup$ What about calculating w_2? Is it just to substitue w_1 into w_2 and calculate the equation which you wrote? $\endgroup$
    – Crispy13
    Dec 28, 2019 at 9:29
  • $\begingroup$ it’s similar, but upto the term h, not g because it’s not in the first layer. $\endgroup$
    – gunes
    Dec 28, 2019 at 9:32
  • $\begingroup$ so $\frac {\partial L}{\partial w_2} = \frac {\partial L}{\partial \hat y}\frac {\partial \hat y}{\partial g}\frac {\partial g}{\partial w_2}$ is right? $\endgroup$
    – Crispy13
    Dec 28, 2019 at 9:36
  • $\begingroup$ Yes, that’s right $\endgroup$
    – gunes
    Dec 28, 2019 at 9:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.