# How chain rule and partial derivative are applied on gradient descent in deep learning?

Suppose there is a DNN like this:

$$h = t(w_1x+b_1)$$
$$g = t(w_2h+b_2)$$
$$\hat y = t(w_3g+b_3)$$
$$Loss = \frac 12\sum (y - \hat y)^2$$

Just one input $$x$$
$$t$$ is activation function.

Could you explain with equations how the gradients of $$w_1$$, $$w_2$$, $$w_3$$ is calculated?

By the way, when combining partial derivative and chain rule,

$$\frac {dk}{dg}\frac {dg}{dh}\frac {dh}{dx}$$ -> $$\frac {\partial k}{\partial g}\frac {\partial g}{\partial h}\frac {\partial h}{\partial w_1}$$

is the above fine?

• how is y related to h,g and k? Dec 28, 2019 at 9:09
• I edited my question. sorry. Dec 28, 2019 at 9:14

We just execute the chain rule: $$\frac{\partial L}{\partial w_1}=\frac{\partial L}{\partial \hat y}\frac{\partial \hat y}{\partial g}\frac{\partial g}{\partial h}\frac{\partial h}{\partial w_1}$$
Then, we calculate each term separately, e.g. $$\frac{\partial h}{\partial w_1}=t'(w_1x+b_1)x,\frac{\partial g}{\partial h}=t'(w_2h+b_2)w_2,\cdots$$
Your chain rule is not fine, because we don't ever calculate $$\frac{\partial h}{\partial x}$$ since $$x$$ is not a parameter nor an outcome.
• so $\frac {\partial L}{\partial w_2} = \frac {\partial L}{\partial \hat y}\frac {\partial \hat y}{\partial g}\frac {\partial g}{\partial w_2}$ is right? Dec 28, 2019 at 9:36