Question with Matrix Derivative: Why do I have to transpose?

Question

In the equation for Recurrent Neural Networks: $$ h_t = \tanh(h_{t-1}W_{hh} + x_tW_{xh} + b) $$

Where $h_t$ is of size (N,H)
Where $W_{hh}$ is of size (H,H)
Where $W_{xh}$ is of size (D,H)
Where $x_t$ is of size (N,D)
Where $b$ is of size (N,H)

I am trying to find the derivative with respect to $h_{t-1}$

Applying chain rule, the result is:
$$ \frac{\partial h_t}{\partial h_{t-1}} = [1-\tanh^2(h_{t-1}W_{hh} + x_tW_{xh} + b)]W_{hh} $$

(Where $1-\tanh^2$ is the derivate of the $\tanh$ function)

However, the correct result requires me to transpose $W_{hh}$ as follows:

$$ \frac{\partial h_t}{\partial h_{t-1}} = [1-\tanh^2(h_{t-1}W_{hh} + x_tW_{xh} + b)]W_{hh}^T $$

I am at a loss of why I need to transpose to get the correct partial ... can someone provide some intuition here?

Any guidance will be greatly appreciated! Thank you!

Answer 1

As noted in the comments, it is best to write out the matrix equations and then apply the standard derivative rules. After a bit of experience with small cases, where you expand out all the terms, you can try just writing out the equations using summations and subscripts.

In terms of mathematical rules, for completeness I will note that there is a pretty good reference called The Matrix Cookbook*, that documents a large number of rules for calculus on matrix equations. (*I am not sure if there is a stable home for this document, so I am linking a Google search, which has always reliably found (many) copies, in my experience!)

That said, I find it difficult to follow these usually, so I end up using summation/subscript forms when I need to compute these sorts of derivatives. (Or sometimes I will use index notation.)

EDIT: For your specific problem, I think you may be getting stuck on the large equations. Here I would suggest that intermediate variables can be your friend, helping to break down the structure (similar to functions in modular programming).

EDIT the second: When I first looked at your problem, I thought the lowercase $h$ variables were vectors. Now I realize they are matrices, which changes things. Given this, it does not make sense to me that the result would be a matrix, rather than a higher-order tensor. The order of a tensor is the number of indices. If you take a derivative of one tensor $U$ with respect to another tensor $V$, the order of the resulting tensor $W$ will be the sum of the two, i.e. $$W_{ij,pq}=\frac{\partial U_{ij}}{\partial V_{pq}}$$

For instance the gradient of a scalar function is a vector ($0+1=1$), and the Jacobian of a vector function is a matrix ($1+1=2$). For an example of a physically-meaningful $4_{th}$ order tensor, a simple example is the "spring constant" for the continuum-mechanics generalization of Hooke's Law (the last equation on that page is essentially the same as the one I give above).

When I work through your case, I find that the result is sparse, so is effectively $3_{rd}$ order, but it is not so sparse as to be $2_{nd}$ order $\ldots$ unless I made a mistake. (More precisely: I find $W_{ij,pq}=0$ for $p\neq i$, so we can effectively work with $\hat{W}_{ij,k}\equiv W_{ij,ik}$ as an order-3 tensor.)

EDIT the third: As the OP requested clarification, here is a simple example to demonstrate the issue. Consider the matrix equation $$U=VA\implies U_{ij}=\sum_kV_{ik}A_{kj}\implies \frac{\partial U_{ij}}{\partial V_{pq}}=\sum_k\frac{\partial V_{ik}}{\partial V_{pq}}A_{kj}$$ To proceed, we introduce the Kronecker delta symbol (essentially the identity matrix): $$\delta_{ij} = \begin{cases} 1 & i=j\\ 0 & i\neq j \end{cases}$$ Then the derivative within the summation can be expressed as $$\frac{\partial V_{ik}}{\partial V_{pq}}=\delta_{ip}\delta_{kq}$$ i.e. it is one if the both indices match and zero otherwise.

Noting the "index replacement" property of a summed tensor-$\delta$ product $$\sum_k\delta_{kq}A_{kj}=A_{qj}$$ we then have $$\frac{\partial U_{ij}}{\partial V_{pq}}=\delta_{ip}A_{qj}$$

So finally $$W_{ij,pq}= \begin{cases} A_{qj} & p=i\\ 0 & p\neq i \end{cases}$$ (and $\hat{W}_{ij,k}=A_{kj}$)