I have a neural network like this

$z_1=W_{1x}\cdot x+b_1$
$z_2=W_2\cdot h_1+W_{2x}\cdot x+b_2$
$y=W_3\cdot h_2+W_{3x}\cdot x+b_3$

input and weights are matrices

Now I want the derivation repect to the input x. I used the chain rule and get:

$W_3 \cdot \text{diag}(\text{RELU'}(h_2)) \cdot (W_2 \cdot \text{diag}(\text{RELU'}(h_1)) \cdot W_{1x}+W_{2x})+W_{3x},$
with $\text{RELU'}(x)=1 \text{ if } x>0, \text{else } 0$

I am very unsure about the derivation, is this correct?

Thanks in advance!

EDIT: if I use the h-derivate like this

$\frac{f(x+h)-f(x)}{h}$ for very small h I get a different result than with my derivation.

My neural network has two inputs, so I use the h-derivate once for $x_1$ and once for $x_2$


1 Answer 1


So, your network looks something like this (renamed some parameters to avoid confusion):

enter image description here

Now: $$ \begin{align*} \frac{dJ}{dx} = &\frac{dJ}{dy} \frac{dy}{dz} \frac{dz}{dh} \frac{dh}{dx}+ \frac{dJ}{dy} \frac{dy}{dz} \frac{dh}{dx}\\ & = \frac{dJ}{dy} \frac{dy}{dz} \frac{dh}{dx} \frac{dz}{dh}+ \frac{dJ}{dy} \frac{dy}{dz} \frac{dz}{dx}\\ & = \frac{dJ}{dy} \frac{dy}{dz} (\frac{dh}{dx} \frac{dz}{dh}+ \frac{dz}{dx})\\ \end{align*} $$

At this point, you are only left with the actual calculations of the single derivatives, which seemed correct in your question... you might just be missing a final ")" (I've also added the derivative of the loss with respect to the output layer, but you might not need it depending on what you are doing with the gradient)

  • $\begingroup$ Did you change h and z in your picture or why is z in front of the output? So you say my derivation is correct, but then I have the following problem: I wanted to check the derivation with the h-derivative and got a different result. $\endgroup$
    – Gaweiliex
    Jul 14, 2022 at 12:28
  • $\begingroup$ @Gaweiliex yes, the names do not correspond... what do you mean with h-derivative? you have some problem in the notation that you have posted, some of those are scalar products, other must be matrix multiplications $\endgroup$
    – Alberto
    Jul 14, 2022 at 15:16
  • $\begingroup$ I Edit my Question for better Visualisation what i mean with h-derivate. You're right with the notation i use the derivative in my Matlab script so this souldn't lead to any issue. $\endgroup$
    – Gaweiliex
    Jul 15, 2022 at 9:16
  • $\begingroup$ I find my mistake! The derivate is right. I had a mistake in my script Thanky You! $\endgroup$
    – Gaweiliex
    Jul 15, 2022 at 10:08
  • $\begingroup$ @Gaweiliex perfect, good luck with your project! $\endgroup$
    – Alberto
    Jul 15, 2022 at 12:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.