1
$\begingroup$

I am trying to understand how back propagation works for a linear layer using minibatches by following this post: https://web.eecs.umich.edu/~justincj/teaching/eecs442/notes/linear-backprop.html.

Imagine the following:

  • The weight matrix, $W$, has shape $D \times M$
  • The input matrix has size $N \times D$, where $N$ is the size of the minibatch

We know the derivative of the loss, $L$, with respect to the output $Y$ of $X \cdot M$ will have dimensions: $N \times M$.

In other words, $\frac{\partial L}{\partial Y}$ has dimensions $N \times M$.

By the chain rule, we know:

$$ \frac{\partial L}{\partial X} = \frac{\partial Y}{\partial X} \frac{\partial L}{\partial Y} $$

where $\frac{\partial Y}{\partial X}$ is a Jacobian matrix.

Here's my issue:

  • We know the dimensions of $\frac{\partial Y}{\partial X}$ will be $N \times M$ by $N \times D$ since there are $N \times M$ outputs and $N \times D$ inputs.
  • The matrix dimensions don't match up for multiplication. $(N \times M) \times (N \times D)$ times a $N \times M$ matrix is not a valid matrix multiplication.

What am I missing?

$\endgroup$
1
  • $\begingroup$ You have mis-stated the chain rule. $\endgroup$
    – whuber
    Jun 23 at 21:49

1 Answer 1

1
$\begingroup$

Using denominator layout notation, the order of the terms in the chain rule is correct. However, the multiplication is undefined because $\frac{\partial Y}{\partial X}$ is a 4D tensor. We don't have any well known rules to multiply a 4D tensor by a 2D tensor (i.e. matrix). The author instead chooses to compute the final derivative element by element and then come up with a formula.

So, I protest the authors about not mentioning

  • the layout notation and the chain rule acc. to the layout chosen
  • tensor dimensionality of the multiplicands and associated issues
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.