# Higher Order of Vectorization in Backpropagation in Neural Network

I have a question about the dimensions of a Jacobian during backpropagation. The network looks like:

The forward propagation can be defined as:

where g is the activation function.

The dimensions of each variable can also be given as:

Now, for back-propagation, by using chain rule, we can get:

To match up with the dimensions, we have:

I know that after applying chain rule, the normal way is to calculate generalized Jacobian matrix and do matrix multiplication. However, the dimension of each part in chain rule above does not match what generalized Jacobian matrix will give us. For example, for the last term in chain rule, the dimension from generalized Jacobian matrix should be (2 X 1) X (2 X 3). However, what course notes say is 1 X 3.

Why is it true?

• I think this formula falls flat when trying to find the gradients, at least for the explanations given currently. Take a look at the last and the second last layer for example: ![enter image description here](i.sstatic.net/cWaNc.gif) is a [2x1] So, to get the gradients of the weights for this layer we will have to perform } ![enter image description here](i.sstatic.net/yMFg0.gif) [1x1] matrix X [2x1] matrix. How is that possible? Am I missing something here I have tried the rest of the layers Commented Feb 12, 2023 at 13:46

You're right that that doesn't make sense as the Jacobian. Furthermore if multiplying jacobians was really how autodiff worked, any pointwise function applied on vector of length $$n$$ would result in a huge $$n \times n$$ Jacobian being created. This is not what happens in any competant autodiff implementation.

In reality, it's not necessary to compute the jacobian in order to perform backpropagation. All that is needed is the "vector jacobian product", or VJP.

If you have a function $$f : \mathbb{R}^n \rightarrow \mathbb{R}^m$$, then $$\text{VJP} : \mathbb{R}^m \times \mathbb{R}^n \rightarrow \mathbb{R}^n$$ is a function which computes $$\text{VJP}(g,x) = J_f(x)^T g$$, where $$g$$ is the incoming gradient vector $$\frac{\partial \mathcal{L}}{\partial f}$$ and $$J_f(x)$$ is the jacobian of $$f$$. Technically this is a JVP rather than VJP but that's just a matter of convention.

The key point is that although one way to implement the VJP is explicitly computing the jacobian and then performing this vector-matrix product, if you are able to compute the VJP without doing that, it is also perfectly fine.

For example, the VJP for $$\sin(x)$$ is just $$\text{VJP}(g,x) = g \circ \cos(x)$$. The VJP of $$f(W, x) = Wx$$ with respect to $$x$$ is simply $$\text{VJP}(g, W, x) = W^Tg$$ and the VJP with respect to $$W$$ is $$\text{VJP}(g, W, x) = gx^T$$

Returning to your question: the expression in 3.30 is actually just computing $$\text{VJP}(g, W, x) = gx^T$$, with all the terms on the RHS except for the right-most being part of $$g$$, and the last term being $$x^T$$.

• Hi shimao. Thanks for your answer. I had the same problem as OP and thanks to you I can see the reasoning of the solution. However, I still don't get where this concept of $VJP(g,x)$ came from. Do you know of any good sources I could read more about it? Thanks. Commented Aug 1, 2019 at 22:10

$$\frac{\partial \mathcal{L}}{\partial W^{[2]}}$$ must be 2x3 as just like dimensions of $$W^{[2]}$$.

I suggest you to use the backprop formulas (and notation) given in Nielsen's book. When the networks gets bigger it is ease to follow

According to that

\begin{align*} \delta^3 &=a^{[3]}-y \\ \delta^2 &= ((W^{[3]^{T}} (a^{[3]}-y)) \odot g'(z^{[2]})) \\ \frac{\partial \mathcal{L}}{\partial w^{[2]}_{jk}} &= a^{[1]}_k \cdot \delta_j^2 \end{align*}

And by going one more step:

\begin{align*} \delta^1 &= ((W^{[2]^{T}} \delta^2 ) \odot g'(z^{[1]})) \\ \frac{\partial \mathcal{L}}{\partial w^{[1]}_{jk}} &= x^{(i)}_k \cdot \delta_j^1 \end{align*}

where $$\delta^1 \in \mathbb{R}^{3\times 1}$$

I hope at least for other people it will be usefull