# Mismatch between the dimensions of Jacobian matrixes when calculating derivatives during backprop?

I am trying to understand how back propagation works for a linear layer using minibatches by following this post: https://web.eecs.umich.edu/~justincj/teaching/eecs442/notes/linear-backprop.html.

Imagine the following:

• The weight matrix, $$W$$, has shape $$D \times M$$
• The input matrix has size $$N \times D$$, where $$N$$ is the size of the minibatch

We know the derivative of the loss, $$L$$, with respect to the output $$Y$$ of $$X \cdot M$$ will have dimensions: $$N \times M$$.

In other words, $$\frac{\partial L}{\partial Y}$$ has dimensions $$N \times M$$.

By the chain rule, we know:

$$\frac{\partial L}{\partial X} = \frac{\partial Y}{\partial X} \frac{\partial L}{\partial Y}$$

where $$\frac{\partial Y}{\partial X}$$ is a Jacobian matrix.

Here's my issue:

• We know the dimensions of $$\frac{\partial Y}{\partial X}$$ will be $$N \times M$$ by $$N \times D$$ since there are $$N \times M$$ outputs and $$N \times D$$ inputs.
• The matrix dimensions don't match up for multiplication. $$(N \times M) \times (N \times D)$$ times a $$N \times M$$ matrix is not a valid matrix multiplication.

What am I missing?

• You have mis-stated the chain rule.
– whuber
Jun 23 at 21:49

## 1 Answer

Using denominator layout notation, the order of the terms in the chain rule is correct. However, the multiplication is undefined because $$\frac{\partial Y}{\partial X}$$ is a 4D tensor. We don't have any well known rules to multiply a 4D tensor by a 2D tensor (i.e. matrix). The author instead chooses to compute the final derivative element by element and then come up with a formula.

So, I protest the authors about not mentioning

• the layout notation and the chain rule acc. to the layout chosen
• tensor dimensionality of the multiplicands and associated issues