7
$\begingroup$

I know that a Jacobian is a matrix holding all of the first-order derivatives for a vector-valued-function. What is the Jacobian of a neural network, though? What are the inputs and what are the outputs of the function to compute the derivative for?

$\endgroup$

3 Answers 3

11
$\begingroup$

A Jacobian is quite a general term indeed. Lets take this simple, single-hidden-layer network $$\hat{\boldsymbol{y}} = g(\mathbf{W}^{(1)} \cdot f(\mathbf{W}^{(0)} \cdot \boldsymbol{x} + \boldsymbol{b}^{(0)}) + \boldsymbol{b}^{(1)})$$ as an example.

When looking around online, I found most people (e.g. here or here) to refer to the weight updates, i.e. $$\frac{\partial \hat{y}}{\partial W^{(0)}_{ij}}, \frac{\partial \hat{y}}{\partial W^{(1)}_{ij}}, \frac{\partial \hat{y}}{\partial b^{(0)}_i}, \frac{\partial \hat{y}}{\partial b^{(1)}_i} $$

In my opinion, though, the Jacobian of a neural network should rather be the Jacobian of the function that is represented by the neural network, i.e. $$net : \mathbb{R}^m \to \mathbb{R}^n : \boldsymbol{x} \mapsto net(\boldsymbol{x}) = g(\mathbf{W}^{(1)} \cdot f(\mathbf{W}^{(0)} \cdot \boldsymbol{x} + \boldsymbol{b}^{(0)}) + \boldsymbol{b}^{(1)}),$$ where $m$ is the dimensionality of the input vectors (the number of features) and $n$ is the dimensionality of the output (the number of classes). The Jacobian of this network would then simply be $\mathbf{J} = \frac{\partial \hat{\boldsymbol{y}}}{\partial{\boldsymbol{x}}}$ with entries $J_{ij} = \frac{\partial \hat{y}_i}{\partial x_j}.$

$\endgroup$
4
$\begingroup$

Classical approach for neural network is to take a batch of samples and calculate average gradient over these samples. For the Jacobian instead of calculating average gradient - you calculate gradient per each sample separately. At the end you end up with matrix that has N rows and M columns, where N is a number of sample propagated through the network and M is total number of parameter in the network. Every row in the Jacobian is the full gradient per individual input sample.

Important to say that computing Jacobian for Neural Network is inefficient in case if you deal with large number of input samples.

$\endgroup$
2
$\begingroup$

Just a graphical representation of what @Mr Tsjolder has said:

The Input for the network is $\mathbf{X}$

and

Let's take the network $\mathbf{O}$=$\mathbf{W}$*$\mathbf{X}$ (We can any other relationship also!! This is just an example!!)

The Jacobian of this network would then simply be $\mathbf{J} = \frac{\partial {\mathbf{O}}}{\partial{\mathbf{X}}}$

enter image description here

$\endgroup$
2
  • 2
    $\begingroup$ This is being automatically flagged as low quality, probably because it is so short. At present it is more of a comment than an answer by our standards. Can you expand on it? We can also turn it into a comment. $\endgroup$
    – Sycorax
    Aug 12, 2021 at 17:57
  • $\begingroup$ Idea was just to show the things just pictorially!! So that the newbies could understand properly where exactly we are having jacobians!! Edited a bit let me know if more explanation is required!! @Sycorax $\endgroup$ Aug 12, 2021 at 18:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.