# Question regarding backpropagation on a minibatch

Below is a simple model with 2 layers and no nonlinearity:

$X$ is a minibatch of vector inputs, $\hat{y}$ is a vector of scalar outputs, $y$ is a vector of scalar responses, $l$ is a scalar loss, and $W$ and $v$ are the parameters.

$$H=XW$$

$$\hat{y}=Hv$$

$$l=\frac{1}{2}(\hat{y}-y)^\intercal(\hat{y}-y)$$

How do I calculate the gradient with respect to $W$?

Applying the chain rule gives (please forgive the abuse of notation):

$$\frac{\partial{l}}{\partial{W}}=\frac{\partial{l}}{\partial{\hat{y}}}\frac{\partial{\hat{y}}}{\partial{H}}\frac{\partial{H}}{\partial{W}}$$

I don't understand the solutions for $\frac{\partial{\hat{y}}}{\partial{H}}$ and $\frac{\partial{H}}{\partial{W}}$. The first is a vector valued function differentiated with respect to a matrix, and the latter is a matrix valued function differentiated with respect to a matrix.

A scalar differentiated with respect to a vector is a gradient vector, a vector differentiated with respect to a vector is a Jacobian matrix, but what is a vector differentiated with respect to a matrix or a matrix differentiated with respect to a matrix?

Wikipedia says that a matrix differentiated with respect to a matrix is a fourth-rank tensor, since it's essentially a matrix where each element is another matrix.

I think I am fundamentally misunderstanding something here.

I am not familiar with derivatives with respect to a vector or a matrix but you can write backpropagation using simpler formulation : one dimensional partial derivatives.

You'd like to write $$\frac{\partial{l}}{\partial{W_{ij}}}= \frac{\partial{l}}{\partial{\hat{y}}} \frac{\partial{\hat{y}}}{\partial{W_{ij}}}$$

But what does $\frac{\partial{l}}{\partial{\hat{y}}}$ means since $\hat{y}$ is not a scalar ? In order to work only with partial derivatives you can write it as

$$\frac{\partial{l}}{\partial{W_{ij}}}= \sum_{k} \frac{\partial{l}}{\partial{\hat{y}_k}} \frac{\partial{\hat{y}_k}}{\partial{W_{ij}}}$$

Since $\hat{y}_k=\sum_{l}H_{kl}v_l$, $\hat{y}_k$ is a function of $\left(H_{k1},H_{k2},...,\right)$ backpropagation follows with $$\frac{\partial{\hat{y}_k}}{\partial{W_{ij}}}= \sum_{l} \frac{\partial{\hat{y}_k}}{\partial{H_{kl}}} \frac{\partial{H_{kl}}}{\partial{W_{ij}}}$$

At that step your backpropagation is over because you can compute $\frac{\partial{l}}{\partial{\hat{y}_k}}$, $\frac{\partial{\hat{y}_k}} {\partial{H_{kl}}}$ and $\frac{\partial{H_{kl}}}{\partial{W_{ij}}}$

• Thank you for clarification on how to calculate the per-entry derivative. I was under the impression that auto-differentiation programs are able to calculate a single expression for $\frac{\partial{l}}{\partial{W}}$ which returns $\frac{\partial{l}}{\partial{W_{ij}}}$ for all $i,j$ pairs simultaneously. If this is true, what would be that expression? Reverse-engineering some examples I've seen I believe the expression will be something resembling $X^\intercal(\hat{y}-y)v^\intercal$, but I want to understand how to get to this. Jul 14 '17 at 20:09