3
$\begingroup$

I know that a Jacobian is a matrix holding all of the first-order derivatives for a vector-valued-function. What is the Jacobian of a neural network, though? What are the inputs and what are the outputs of the function to compute the derivative for?

$\endgroup$
3
$\begingroup$

Classical approach for neural network is to take a batch of samples and calculate average gradient over these samples. For the Jacobian instead of calculating average gradient - you calculate gradient per each sample separately. At the end you end up with matrix that has N rows and M columns, where N is a number of sample propagated through the network and M is total number of parameter in the network. Every row in the Jacobian is the full gradient per individual input sample.

Important to say that computing Jacobian for Neural Network is inefficient in case if you deal with large number of input samples.

$\endgroup$
3
$\begingroup$

A Jacobian is quite a general term indeed. Lets take this simple, single-hidden-layer network $$\hat{\boldsymbol{y}} = g(\mathbf{W}^{(1)} \cdot f(\mathbf{W}^{(0)} \cdot \boldsymbol{x} + \boldsymbol{b}^{(0)}) + \boldsymbol{b}^{(1)})$$ as an example.

When looking around online, I found most people (e.g. here or here) to refer to the weight updates, i.e. $$\frac{\partial \hat{y}}{\partial W^{(0)}_{ij}}, \frac{\partial \hat{y}}{\partial W^{(1)}_{ij}}, \frac{\partial \hat{y}}{\partial b^{(0)}_i}, \frac{\partial \hat{y}}{\partial b^{(1)}_i} $$

In my opinion, though, the Jacobian of a neural network should rather be the Jacobian of the function that is represented by the neural network, i.e. $$net : \mathbb{R}^m \to \mathbb{R}^n : \boldsymbol{x} \mapsto net(\boldsymbol{x}) = g(\mathbf{W}^{(1)} \cdot f(\mathbf{W}^{(0)} \cdot \boldsymbol{x} + \boldsymbol{b}^{(0)}) + \boldsymbol{b}^{(1)}),$$ where $m$ is the dimensionality of the input vectors (the number of features) and $n$ is the dimensionality of the output (the number of classes). The Jacobian of this network would then simply be $\mathbf{J} = \frac{\partial \hat{\boldsymbol{y}}}{\partial{\boldsymbol{x}}}$ with entries $J_{ij} = \frac{\partial \hat{y}_i}{\partial x_j}.$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.