# Why do derivative of weights for sigmoid binary output classification layer depend on the previous layer?

I've seen the derivative of the weights for a sigmoid binary output classification layer written as:

$\frac{\partial \mathcal{J}}{\partial W^{[i]}} = \frac{1}{m}\frac{\partial \mathcal{J}}{\partial Z^{[i]}}A^{[i-1]T}$

Where

$A^{[i]} = \sigma(Z^{[i]}) \\ Z^{[i]} = W^{[i]}A^{[i-1]} + b^{[i]} \\ \sigma(x) = \frac{1}{1 + e^{-x}} \\ \\ \mathcal{J}(W, b) = \frac{1}{m} \sum\limits_{i = 1}^{m} \mathcal{L}(\hat{y}_i, y_i) \\ \mathcal{L}(\hat{y}, y) = - \Big(y \log \hat{y} + (1-y)\log(1 - \hat{y})\Big)$

Why is the term $A^{[i-1]}$ used here? I would naively assume that the derivative of the weights for the last layer would only depend on the last layer, and not prior layers since the gradients are propagated backwards.

• I believe that I answered this or a similar question some time ago. These questions pop up with regularity synced with ML MOOC session starts in Coursera and Udemy. The answer is in chain rule of differentiation. The only that gets in a way of understanding is that the weights and inputs are vectors and matrices. – Aksakal Feb 13 '18 at 19:44
• @Aksakal, indeed I am doing a MOOC. If possible, could you point me your answer? I've tried searching your history but couldn't find it - stats.stackexchange.com/search?q=user%3A36041+derivative – Michael Barton Feb 13 '18 at 21:29
• It's this answer, not the same but a similar question, I believe. Otherwise, if you posted the definitions of the Z and A, you'd get a better chance of someone helping you, because not everyone's into NN, but everyone knows the likelihoods and loss functions and gradients – Aksakal Feb 13 '18 at 21:33
• Thank you referring your other answer. I've updated with the specifications of Z and A too. – Michael Barton Feb 13 '18 at 21:50
• Add your loss function too, that's where $1/m$ is coming from, from the sum of losses of samples – Aksakal Feb 13 '18 at 22:13

The simplest way to build the intuition is to forget about the sigmoids and vectors for a second. Think of a linear single neuron: its input $x$ is the output of the previous layer $a[i-1]$, and the output is $a[i]=w[i]x+b=w[i]a[i-1]+b$
If you take the derivative of the output with respect to its weight, then you get the following: $$\frac{\partial a[i]}{\partial w[i]}=x=a[i-1]$$
• Thank you, this makes sense to me I think. Is that because the derivative of w becomes $1 \cdot w^0 = 1$? Sorry if that is naive question, I am learning the calculus at the same time. – Michael Barton Feb 15 '18 at 22:03
• It's straight calculus, just replace weight with x if it makes it easier for you, like $(ax)'=a$, in your case it's $w[i]$ instead of $x$ – Aksakal Feb 15 '18 at 22:59