I am looking for a general closed-form formula for the derivatives of a Feed-forward Network with respect to the inputs.

Mathematically, we can write:

$$ \mathbf{y} = f_{FF}(\mathbf{x}) = \mathbf{W}_{h} (\sigma_{h-1}(\mathbf{W}_{h-1} (\ldots \sigma_0(\mathbf{W}_0 \mathbf{x} + \mathbf{b}_0) \ldots) + \mathbf{b}_{h-1})) + \mathbf{b}_{h} $$

Assuming all the activation functions being identical and equal to the $\tanh$, that is $\sigma_h(\cdot) = \tanh(\cdot)$, is there a general closed-formula for the $k$-th order derivatives of $\mathbf{y} = [y_0, \ldots, y_j]$ with respect to $\mathbf{x} = [x_0, \ldots, x_i]$?:

$$ \frac{\partial^k \mathbf{y}^i}{\partial {x^i}^k} = ? $$

This question is a generalization of Neural network derivative with respect to input



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.