Derivate of Neural Network respect to input

I have a neural network like this

$$x=\text{input}$$
$$z_1=W_{1x}\cdot x+b_1$$
$$h_1=\text{relu}(z_1)$$
$$z_2=W_2\cdot h_1+W_{2x}\cdot x+b_2$$
$$h_2=\text{relu}(z_2)$$
$$y=W_3\cdot h_2+W_{3x}\cdot x+b_3$$

input and weights are matrices

Now I want the derivation repect to the input x. I used the chain rule and get:

$$W_3 \cdot \text{diag}(\text{RELU'}(h_2)) \cdot (W_2 \cdot \text{diag}(\text{RELU'}(h_1)) \cdot W_{1x}+W_{2x})+W_{3x},$$
with $$\text{RELU'}(x)=1 \text{ if } x>0, \text{else } 0$$

I am very unsure about the derivation, is this correct?

EDIT: if I use the h-derivate like this

$$\frac{f(x+h)-f(x)}{h}$$ for very small h I get a different result than with my derivation.

My neural network has two inputs, so I use the h-derivate once for $$x_1$$ and once for $$x_2$$

Now: \begin{align*} \frac{dJ}{dx} = &\frac{dJ}{dy} \frac{dy}{dz} \frac{dz}{dh} \frac{dh}{dx}+ \frac{dJ}{dy} \frac{dy}{dz} \frac{dh}{dx}\\ & = \frac{dJ}{dy} \frac{dy}{dz} \frac{dh}{dx} \frac{dz}{dh}+ \frac{dJ}{dy} \frac{dy}{dz} \frac{dz}{dx}\\ & = \frac{dJ}{dy} \frac{dy}{dz} (\frac{dh}{dx} \frac{dz}{dh}+ \frac{dz}{dx})\\ \end{align*}