2
$\begingroup$

My question is how to differentiate a feedforward neural network with respect to an input? I have a network with two hidden layers: the first one has 20 neurons, the second one - 10. The output is a single number.

Mathematically we can write it as follows:

$$ \displaystyle y = W_{out}\cdot f\left(W_2\cdot f\left(W_1\cdot x_0+b_1\right)+b_2\right)+b_{out} $$

Applying the chain rule I got a derivative with respect to i-th element of the input vector:

$$ \frac{\partial y}{\partial x_0^i} = W_{out}\cdot f'(W_2\cdot f(W_1x_0+b_1)+b_2)\cdot W_2\cdot f'(W_1\cdot x_0+b_1)\cdot\frac{\partial W_1x_0}{\partial x_0^i} $$

So the question is: where is my mistake?*

This derivative has a dimension equal to $20\times1$**, but must be a scalar.


*: maybe I'm just really tired and the mistake is really stupid... If it is so, I'm sorry :)

**: because $\frac{\partial W_1x_0}{\partial x_0^i}$ has a dimension equal to $20\times1$: $$ \frac{\partial W_1x_0}{\partial x_0^i} = \begin{bmatrix} W_{1,i}^1 \\ W_{2,i}^1 \\ \vdots \\ W_{20,i}^1 \end{bmatrix} $$ where $W_{1,i}^1$ stands for the first element of i-th column in matrix $W_1$

$\endgroup$
1
  • 1
    $\begingroup$ Have a look at this. $\endgroup$
    – mhdadk
    Jan 16, 2021 at 6:13

1 Answer 1

1
$\begingroup$

I found my mistake: the $f$ function is actually applied element-wise, but I did a differentiation as it was applied to matrix directly. The correct solution is as follows:

Consider $f[...]$ to be equal to $f(W_2\cdot f(W_1\cdot x_0 + b_1)+b_2)$ and $f(...)$ to be equal to $f(W_1\cdot x_0 + b_1)$, so: $$y=\sum_{i=1}^{10} W_{out}^i f[...]^i$$ $$\frac{\partial}{\partial x_0^k}y=\sum_{i=1}^{10}W_{out}^i \frac{\partial}{\partial x_0^k}f[...]^i$$ $$f[...]^i=f(\sum_{j=1}^{20} W_{i,j}^2f(\sum_{n=1}^{52} W_{1,n}^1 x_0^n + b_1^j) + b_2^i)$$ $$\frac{\partial}{\partial x_0^k}f[...]^i=f'[...]^i\sum_{j=1}^{20}W_{i,j}^2f(...)^jW_{j,k}^1$$ $$\frac{\partial}{\partial x_0^k}y=\sum_{i=1}^{10}W_{out}^i f'[...]^i\sum_{j=1}^{20}W_{i,j}^2f'(...)^jW_{j,k}^1$$

$a^i$ stands for i-th element of vector a. This answer can have some computational mistakes, but I suppose, it's correct at least semantically

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.