# Why do derivative of weights for sigmoid binary output classification layer depend on the previous layer?

I've seen the derivative of the weights for a sigmoid binary output classification layer written as:

$\frac{\partial \mathcal{J}}{\partial W^{[i]}} = \frac{1}{m}\frac{\partial \mathcal{J}}{\partial Z^{[i]}}A^{[i-1]T}$

Where

$A^{[i]} = \sigma(Z^{[i]}) \\ Z^{[i]} = W^{[i]}A^{[i-1]} + b^{[i]} \\ \sigma(x) = \frac{1}{1 + e^{-x}} \\ \\ \mathcal{J}(W, b) = \frac{1}{m} \sum\limits_{i = 1}^{m} \mathcal{L}(\hat{y}_i, y_i) \\ \mathcal{L}(\hat{y}, y) = - \Big(y \log \hat{y} + (1-y)\log(1 - \hat{y})\Big)$

Why is the term $A^{[i-1]}$ used here? I would naively assume that the derivative of the weights for the last layer would only depend on the last layer, and not prior layers since the gradients are propagated backwards.

• I believe that I answered this or a similar question some time ago. These questions pop up with regularity synced with ML MOOC session starts in Coursera and Udemy. The answer is in chain rule of differentiation. The only that gets in a way of understanding is that the weights and inputs are vectors and matrices. Feb 13, 2018 at 19:44
• @Aksakal, indeed I am doing a MOOC. If possible, could you point me your answer? I've tried searching your history but couldn't find it - stats.stackexchange.com/search?q=user%3A36041+derivative Feb 13, 2018 at 21:29
• It's this answer, not the same but a similar question, I believe. Otherwise, if you posted the definitions of the Z and A, you'd get a better chance of someone helping you, because not everyone's into NN, but everyone knows the likelihoods and loss functions and gradients Feb 13, 2018 at 21:33
• Thank you referring your other answer. I've updated with the specifications of Z and A too. Feb 13, 2018 at 21:50
• Add your loss function too, that's where $1/m$ is coming from, from the sum of losses of samples Feb 13, 2018 at 22:13

The simplest way to build the intuition is to forget about the sigmoids and vectors for a second. Think of a linear single neuron: its input $x$ is the output of the previous layer $a[i-1]$, and the output is $a[i]=w[i]x+b=w[i]a[i-1]+b$
If you take the derivative of the output with respect to its weight, then you get the following: $$\frac{\partial a[i]}{\partial w[i]}=x=a[i-1]$$
• Thank you, this makes sense to me I think. Is that because the derivative of w becomes $1 \cdot w^0 = 1$? Sorry if that is naive question, I am learning the calculus at the same time. Feb 15, 2018 at 22:03
• It's straight calculus, just replace weight with x if it makes it easier for you, like $(ax)'=a$, in your case it's $w[i]$ instead of $x$ Feb 15, 2018 at 22:59