Derivate of Neural Network respect to input

Question

I have a neural network like this

$x=\text{input}$
$z_1=W_{1x}\cdot x+b_1$
$h_1=\text{relu}(z_1)$
$z_2=W_2\cdot h_1+W_{2x}\cdot x+b_2$
$h_2=\text{relu}(z_2)$
$y=W_3\cdot h_2+W_{3x}\cdot x+b_3$

input and weights are matrices

Now I want the derivation repect to the input x. I used the chain rule and get:

$W_3 \cdot \text{diag}(\text{RELU'}(h_2)) \cdot (W_2 \cdot \text{diag}(\text{RELU'}(h_1)) \cdot W_{1x}+W_{2x})+W_{3x},$
with $\text{RELU'}(x)=1 \text{ if } x>0, \text{else } 0$

I am very unsure about the derivation, is this correct?

Thanks in advance!

EDIT: if I use the h-derivate like this

$\frac{f(x+h)-f(x)}{h}$ for very small h I get a different result than with my derivation.

My neural network has two inputs, so I use the h-derivate once for $x_1$ and once for $x_2$

Answer 1

So, your network looks something like this (renamed some parameters to avoid confusion):

Now: $$ \begin{align*} \frac{dJ}{dx} = &\frac{dJ}{dy} \frac{dy}{dz} \frac{dz}{dh} \frac{dh}{dx}+ \frac{dJ}{dy} \frac{dy}{dz} \frac{dh}{dx}\\ & = \frac{dJ}{dy} \frac{dy}{dz} \frac{dh}{dx} \frac{dz}{dh}+ \frac{dJ}{dy} \frac{dy}{dz} \frac{dz}{dx}\\ & = \frac{dJ}{dy} \frac{dy}{dz} (\frac{dh}{dx} \frac{dz}{dh}+ \frac{dz}{dx})\\ \end{align*} $$

At this point, you are only left with the actual calculations of the single derivatives, which seemed correct in your question... you might just be missing a final ")" (I've also added the derivative of the loss with respect to the output layer, but you might not need it depending on what you are doing with the gradient)