Question regarding backpropagation on a minibatch Below is a simple model with 2 layers and no nonlinearity:
$X$ is a minibatch of vector inputs, $\hat{y}$ is a vector of scalar outputs, $y$ is a vector of scalar responses, $l$ is a scalar loss, and $W$ and $v$ are the parameters.
$$H=XW$$
$$\hat{y}=Hv$$
$$l=\frac{1}{2}(\hat{y}-y)^\intercal(\hat{y}-y)$$
How do I calculate the gradient with respect to $W$?
Applying the chain rule gives (please forgive the abuse of notation):
$$\frac{\partial{l}}{\partial{W}}=\frac{\partial{l}}{\partial{\hat{y}}}\frac{\partial{\hat{y}}}{\partial{H}}\frac{\partial{H}}{\partial{W}}$$
I don't understand the solutions for $\frac{\partial{\hat{y}}}{\partial{H}}$ and $\frac{\partial{H}}{\partial{W}}$. The first is a vector valued function differentiated with respect to a matrix, and the latter is a matrix valued function differentiated with respect to a matrix.
A scalar differentiated with respect to a vector is a gradient vector, a vector differentiated with respect to a vector is a Jacobian matrix, but what is a vector differentiated with respect to a matrix or a matrix differentiated with respect to a matrix?
Wikipedia says that a matrix differentiated with respect to a matrix is a fourth-rank tensor, since it's essentially a matrix where each element is another matrix.
I think I am fundamentally misunderstanding something here.
 A: I am not familiar with derivatives with respect to a vector or a matrix but you can write backpropagation using simpler formulation : one dimensional partial derivatives. 
You'd like to write 
$$\frac{\partial{l}}{\partial{W_{ij}}}=
\frac{\partial{l}}{\partial{\hat{y}}}
\frac{\partial{\hat{y}}}{\partial{W_{ij}}}$$
But what does $\frac{\partial{l}}{\partial{\hat{y}}}$ means since $\hat{y}$ is not a scalar ? 
In order to work only with partial derivatives you can write it as 
$$\frac{\partial{l}}{\partial{W_{ij}}}=
\sum_{k}
\frac{\partial{l}}{\partial{\hat{y}_k}}
\frac{\partial{\hat{y}_k}}{\partial{W_{ij}}}$$
Since $\hat{y}_k=\sum_{l}H_{kl}v_l$, $\hat{y}_k$ is a function of $\left(H_{k1},H_{k2},...,\right)$ backpropagation follows with
$$\frac{\partial{\hat{y}_k}}{\partial{W_{ij}}}=
\sum_{l}
\frac{\partial{\hat{y}_k}}{\partial{H_{kl}}}
\frac{\partial{H_{kl}}}{\partial{W_{ij}}}$$
At that step your backpropagation is over because you can compute $\frac{\partial{l}}{\partial{\hat{y}_k}}$, $\frac{\partial{\hat{y}_k}} {\partial{H_{kl}}}$ and $\frac{\partial{H_{kl}}}{\partial{W_{ij}}}$
