# Neural network derivative with respect to input

My question is how to differentiate a feedforward neural network with respect to an input? I have a network with two hidden layers: the first one has 20 neurons, the second one - 10. The output is a single number.

Mathematically we can write it as follows:

$$\displaystyle y = W_{out}\cdot f\left(W_2\cdot f\left(W_1\cdot x_0+b_1\right)+b_2\right)+b_{out}$$

Applying the chain rule I got a derivative with respect to i-th element of the input vector:

$$\frac{\partial y}{\partial x_0^i} = W_{out}\cdot f'(W_2\cdot f(W_1x_0+b_1)+b_2)\cdot W_2\cdot f'(W_1\cdot x_0+b_1)\cdot\frac{\partial W_1x_0}{\partial x_0^i}$$

So the question is: where is my mistake?*

This derivative has a dimension equal to $$20\times1$$**, but must be a scalar.

*: maybe I'm just really tired and the mistake is really stupid... If it is so, I'm sorry :)

**: because $$\frac{\partial W_1x_0}{\partial x_0^i}$$ has a dimension equal to $$20\times1$$: $$\frac{\partial W_1x_0}{\partial x_0^i} = \begin{bmatrix} W_{1,i}^1 \\ W_{2,i}^1 \\ \vdots \\ W_{20,i}^1 \end{bmatrix}$$ where $$W_{1,i}^1$$ stands for the first element of i-th column in matrix $$W_1$$

• Have a look at this. Commented Jan 16, 2021 at 6:13

I found my mistake: the $$f$$ function is actually applied element-wise, but I did a differentiation as it was applied to matrix directly. The correct solution is as follows:
Consider $$f[...]$$ to be equal to $$f(W_2\cdot f(W_1\cdot x_0 + b_1)+b_2)$$ and $$f(...)$$ to be equal to $$f(W_1\cdot x_0 + b_1)$$, so: $$y=\sum_{i=1}^{10} W_{out}^i f[...]^i$$ $$\frac{\partial}{\partial x_0^k}y=\sum_{i=1}^{10}W_{out}^i \frac{\partial}{\partial x_0^k}f[...]^i$$ $$f[...]^i=f(\sum_{j=1}^{20} W_{i,j}^2f(\sum_{n=1}^{52} W_{1,n}^1 x_0^n + b_1^j) + b_2^i)$$ $$\frac{\partial}{\partial x_0^k}f[...]^i=f'[...]^i\sum_{j=1}^{20}W_{i,j}^2f(...)^jW_{j,k}^1$$ $$\frac{\partial}{\partial x_0^k}y=\sum_{i=1}^{10}W_{out}^i f'[...]^i\sum_{j=1}^{20}W_{i,j}^2f'(...)^jW_{j,k}^1$$
$$a^i$$ stands for i-th element of vector a. This answer can have some computational mistakes, but I suppose, it's correct at least semantically