Neural network derivative with respect to input

Question

My question is how to differentiate a feedforward neural network with respect to an input? I have a network with two hidden layers: the first one has 20 neurons, the second one - 10. The output is a single number.

Mathematically we can write it as follows:

$$ \displaystyle y = W_{out}\cdot f\left(W_2\cdot f\left(W_1\cdot x_0+b_1\right)+b_2\right)+b_{out} $$

Applying the chain rule I got a derivative with respect to i-th element of the input vector:

$$ \frac{\partial y}{\partial x_0^i} = W_{out}\cdot f'(W_2\cdot f(W_1x_0+b_1)+b_2)\cdot W_2\cdot f'(W_1\cdot x_0+b_1)\cdot\frac{\partial W_1x_0}{\partial x_0^i} $$

So the question is: where is my mistake?*

This derivative has a dimension equal to $20\times1$**, but must be a scalar.

---

*: maybe I'm just really tired and the mistake is really stupid... If it is so, I'm sorry :)

**: because $\frac{\partial W_1x_0}{\partial x_0^i}$ has a dimension equal to $20\times1$: $$ \frac{\partial W_1x_0}{\partial x_0^i} = \begin{bmatrix} W_{1,i}^1 \\ W_{2,i}^1 \\ \vdots \\ W_{20,i}^1 \end{bmatrix} $$ where $W_{1,i}^1$ stands for the first element of i-th column in matrix $W_1$

Answer 1

I found my mistake: the $f$ function is actually applied element-wise, but I did a differentiation as it was applied to matrix directly. The correct solution is as follows:

Consider $f[...]$ to be equal to $f(W_2\cdot f(W_1\cdot x_0 + b_1)+b_2)$ and $f(...)$ to be equal to $f(W_1\cdot x_0 + b_1)$, so: $$y=\sum_{i=1}^{10} W_{out}^i f[...]^i$$ $$\frac{\partial}{\partial x_0^k}y=\sum_{i=1}^{10}W_{out}^i \frac{\partial}{\partial x_0^k}f[...]^i$$ $$f[...]^i=f(\sum_{j=1}^{20} W_{i,j}^2f(\sum_{n=1}^{52} W_{1,n}^1 x_0^n + b_1^j) + b_2^i)$$ $$\frac{\partial}{\partial x_0^k}f[...]^i=f'[...]^i\sum_{j=1}^{20}W_{i,j}^2f(...)^jW_{j,k}^1$$ $$\frac{\partial}{\partial x_0^k}y=\sum_{i=1}^{10}W_{out}^i f'[...]^i\sum_{j=1}^{20}W_{i,j}^2f'(...)^jW_{j,k}^1$$

$a^i$ stands for i-th element of vector a. This answer can have some computational mistakes, but I suppose, it's correct at least semantically