I've seen the derivative of the weights for a sigmoid binary output classification layer written as:

$\frac{\partial \mathcal{J}}{\partial W^{[i]}} = \frac{1}{m}\frac{\partial \mathcal{J}}{\partial Z^{[i]}}A^{[i-1]T}$


$A^{[i]} = \sigma(Z^{[i]}) \\ Z^{[i]} = W^{[i]}A^{[i-1]} + b^{[i]} \\ \sigma(x) = \frac{1}{1 + e^{-x}} \\ \\ \mathcal{J}(W, b) = \frac{1}{m} \sum\limits_{i = 1}^{m} \mathcal{L}(\hat{y}_i, y_i) \\ \mathcal{L}(\hat{y}, y) = - \Big(y \log \hat{y} + (1-y)\log(1 - \hat{y})\Big) $

Why is the term $A^{[i-1]}$ used here? I would naively assume that the derivative of the weights for the last layer would only depend on the last layer, and not prior layers since the gradients are propagated backwards.

  • 3
    $\begingroup$ I believe that I answered this or a similar question some time ago. These questions pop up with regularity synced with ML MOOC session starts in Coursera and Udemy. The answer is in chain rule of differentiation. The only that gets in a way of understanding is that the weights and inputs are vectors and matrices. $\endgroup$ – Aksakal Feb 13 '18 at 19:44
  • $\begingroup$ @Aksakal, indeed I am doing a MOOC. If possible, could you point me your answer? I've tried searching your history but couldn't find it - stats.stackexchange.com/search?q=user%3A36041+derivative $\endgroup$ – Michael Barton Feb 13 '18 at 21:29
  • $\begingroup$ It's this answer, not the same but a similar question, I believe. Otherwise, if you posted the definitions of the Z and A, you'd get a better chance of someone helping you, because not everyone's into NN, but everyone knows the likelihoods and loss functions and gradients $\endgroup$ – Aksakal Feb 13 '18 at 21:33
  • $\begingroup$ Thank you referring your other answer. I've updated with the specifications of Z and A too. $\endgroup$ – Michael Barton Feb 13 '18 at 21:50
  • $\begingroup$ Add your loss function too, that's where $1/m$ is coming from, from the sum of losses of samples $\endgroup$ – Aksakal Feb 13 '18 at 22:13

The simplest way to build the intuition is to forget about the sigmoids and vectors for a second. Think of a linear single neuron: its input $x$ is the output of the previous layer $a[i-1]$, and the output is $a[i]=w[i]x+b=w[i]a[i-1]+b$

If you take the derivative of the output with respect to its weight, then you get the following: $$\frac{\partial a[i]}{\partial w[i]}=x=a[i-1]$$

This is where the previous layer's output comes from. If you take the derivative of the loss, then recursively through the chain rule at some point you'll end up taking a derivative of the output of this layer with respect to its weight, and the previous layer's output will pop up.

  • $\begingroup$ Thank you, this makes sense to me I think. Is that because the derivative of w becomes $1 \cdot w^0 = 1$? Sorry if that is naive question, I am learning the calculus at the same time. $\endgroup$ – Michael Barton Feb 15 '18 at 22:03
  • $\begingroup$ It's straight calculus, just replace weight with x if it makes it easier for you, like $(ax)'=a$, in your case it's $w[i]$ instead of $x$ $\endgroup$ – Aksakal Feb 15 '18 at 22:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.