What is the jacobian for a neural network

Question

I know that a Jacobian is a matrix holding all of the first-order derivatives for a vector-valued-function. What is the Jacobian of a neural network, though? What are the inputs and what are the outputs of the function to compute the derivative for?

Answer 1

A Jacobian is quite a general term indeed. Lets take this simple, single-hidden-layer network $$\hat{\boldsymbol{y}} = g(\mathbf{W}^{(1)} \cdot f(\mathbf{W}^{(0)} \cdot \boldsymbol{x} + \boldsymbol{b}^{(0)}) + \boldsymbol{b}^{(1)})$$ as an example.

When looking around online, I found most people (e.g. here or here) to refer to the weight updates, i.e. $$\frac{\partial \hat{y}}{\partial W^{(0)}_{ij}}, \frac{\partial \hat{y}}{\partial W^{(1)}_{ij}}, \frac{\partial \hat{y}}{\partial b^{(0)}_i}, \frac{\partial \hat{y}}{\partial b^{(1)}_i} $$

In my opinion, though, the Jacobian of a neural network should rather be the Jacobian of the function that is represented by the neural network, i.e. $$net : \mathbb{R}^m \to \mathbb{R}^n : \boldsymbol{x} \mapsto net(\boldsymbol{x}) = g(\mathbf{W}^{(1)} \cdot f(\mathbf{W}^{(0)} \cdot \boldsymbol{x} + \boldsymbol{b}^{(0)}) + \boldsymbol{b}^{(1)}),$$ where $m$ is the dimensionality of the input vectors (the number of features) and $n$ is the dimensionality of the output (the number of classes). The Jacobian of this network would then simply be $\mathbf{J} = \frac{\partial \hat{\boldsymbol{y}}}{\partial{\boldsymbol{x}}}$ with entries $J_{ij} = \frac{\partial \hat{y}_i}{\partial x_j}.$

Answer 2

Classical approach for neural network is to take a batch of samples and calculate average gradient over these samples. For the Jacobian instead of calculating average gradient - you calculate gradient per each sample separately. At the end you end up with matrix that has N rows and M columns, where N is a number of sample propagated through the network and M is total number of parameter in the network. Every row in the Jacobian is the full gradient per individual input sample.

Important to say that computing Jacobian for Neural Network is inefficient in case if you deal with large number of input samples.

Answer 3

Just a graphical representation of what @Mr Tsjolder has said:

The Input for the network is $\mathbf{X}$

and

Let's take the network $\mathbf{O}$=$\mathbf{W}$*$\mathbf{X}$ (We can any other relationship also!! This is just an example!!)

The Jacobian of this network would then simply be $\mathbf{J} = \frac{\partial {\mathbf{O}}}{\partial{\mathbf{X}}}$