1
$\begingroup$

Below is a simple model with 2 layers and no nonlinearity:

$X$ is a minibatch of vector inputs, $\hat{y}$ is a vector of scalar outputs, $y$ is a vector of scalar responses, $l$ is a scalar loss, and $W$ and $v$ are the parameters.

$$H=XW$$

$$\hat{y}=Hv$$

$$l=\frac{1}{2}(\hat{y}-y)^\intercal(\hat{y}-y)$$

How do I calculate the gradient with respect to $W$?

Applying the chain rule gives (please forgive the abuse of notation):

$$\frac{\partial{l}}{\partial{W}}=\frac{\partial{l}}{\partial{\hat{y}}}\frac{\partial{\hat{y}}}{\partial{H}}\frac{\partial{H}}{\partial{W}}$$

I don't understand the solutions for $\frac{\partial{\hat{y}}}{\partial{H}}$ and $\frac{\partial{H}}{\partial{W}}$. The first is a vector valued function differentiated with respect to a matrix, and the latter is a matrix valued function differentiated with respect to a matrix.

A scalar differentiated with respect to a vector is a gradient vector, a vector differentiated with respect to a vector is a Jacobian matrix, but what is a vector differentiated with respect to a matrix or a matrix differentiated with respect to a matrix?

Wikipedia says that a matrix differentiated with respect to a matrix is a fourth-rank tensor, since it's essentially a matrix where each element is another matrix.

I think I am fundamentally misunderstanding something here.

$\endgroup$
2
$\begingroup$

I am not familiar with derivatives with respect to a vector or a matrix but you can write backpropagation using simpler formulation : one dimensional partial derivatives.

You'd like to write $$\frac{\partial{l}}{\partial{W_{ij}}}= \frac{\partial{l}}{\partial{\hat{y}}} \frac{\partial{\hat{y}}}{\partial{W_{ij}}}$$

But what does $\frac{\partial{l}}{\partial{\hat{y}}}$ means since $\hat{y}$ is not a scalar ? In order to work only with partial derivatives you can write it as

$$\frac{\partial{l}}{\partial{W_{ij}}}= \sum_{k} \frac{\partial{l}}{\partial{\hat{y}_k}} \frac{\partial{\hat{y}_k}}{\partial{W_{ij}}}$$

Since $\hat{y}_k=\sum_{l}H_{kl}v_l$, $\hat{y}_k$ is a function of $\left(H_{k1},H_{k2},...,\right)$ backpropagation follows with $$\frac{\partial{\hat{y}_k}}{\partial{W_{ij}}}= \sum_{l} \frac{\partial{\hat{y}_k}}{\partial{H_{kl}}} \frac{\partial{H_{kl}}}{\partial{W_{ij}}}$$

At that step your backpropagation is over because you can compute $\frac{\partial{l}}{\partial{\hat{y}_k}}$, $\frac{\partial{\hat{y}_k}} {\partial{H_{kl}}}$ and $\frac{\partial{H_{kl}}}{\partial{W_{ij}}}$

$\endgroup$
  • $\begingroup$ Thank you for clarification on how to calculate the per-entry derivative. I was under the impression that auto-differentiation programs are able to calculate a single expression for $\frac{\partial{l}}{\partial{W}}$ which returns $\frac{\partial{l}}{\partial{W_{ij}}}$ for all $i,j$ pairs simultaneously. If this is true, what would be that expression? Reverse-engineering some examples I've seen I believe the expression will be something resembling $X^\intercal(\hat{y}-y)v^\intercal$, but I want to understand how to get to this. $\endgroup$ – tmakino Jul 14 '17 at 20:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.