# Question with Matrix Derivative: Why do I have to transpose?

In the equation for Recurrent Neural Networks: $$h_t = \tanh(h_{t-1}W_{hh} + x_tW_{xh} + b)$$

Where $h_t$ is of size (N,H)
Where $W_{hh}$ is of size (H,H)
Where $W_{xh}$ is of size (D,H)
Where $x_t$ is of size (N,D)
Where $b$ is of size (N,H)

I am trying to find the derivative with respect to $h_{t-1}$

Applying chain rule, the result is:
$$\frac{\partial h_t}{\partial h_{t-1}} = [1-\tanh^2(h_{t-1}W_{hh} + x_tW_{xh} + b)]W_{hh}$$

(Where $1-\tanh^2$ is the derivate of the $\tanh$ function)

However, the correct result requires me to transpose $W_{hh}$ as follows:

$$\frac{\partial h_t}{\partial h_{t-1}} = [1-\tanh^2(h_{t-1}W_{hh} + x_tW_{xh} + b)]W_{hh}^T$$

I am at a loss of why I need to transpose to get the correct partial ... can someone provide some intuition here?

Any guidance will be greatly appreciated! Thank you!

• Write it out for the simplest revealing case where $N=1,H=2$.
– whuber
Sep 8, 2016 at 17:45
• I can see that the solution is correct, however, I am trying to understand what mathematical rule I am missing so I can derive future more complex equations. Sep 8, 2016 at 17:54
• You will appreciate that rule by writing things out for small, simple cases.
– whuber
Sep 8, 2016 at 18:01
• I did write it out but I didn't discover anything. The only thing I could possibly see is the order of the multiplication: $W_{hh}$ *$dtanh$ vs $dtanh$*$W_{hh}^T$ .. is this possibly in the right track? @whuber Sep 8, 2016 at 18:30
• I started writing an answer but couldn't figure out notation. Are you taking $\mathrm{tanh}$ to act elementwise on the $N\times H$ matrix arguments?
– KOE
Sep 8, 2016 at 23:45

## 1 Answer

As noted in the comments, it is best to write out the matrix equations and then apply the standard derivative rules. After a bit of experience with small cases, where you expand out all the terms, you can try just writing out the equations using summations and subscripts.

In terms of mathematical rules, for completeness I will note that there is a pretty good reference called The Matrix Cookbook*, that documents a large number of rules for calculus on matrix equations. (*I am not sure if there is a stable home for this document, so I am linking a Google search, which has always reliably found (many) copies, in my experience!)

That said, I find it difficult to follow these usually, so I end up using summation/subscript forms when I need to compute these sorts of derivatives. (Or sometimes I will use index notation.)

EDIT: For your specific problem, I think you may be getting stuck on the large equations. Here I would suggest that intermediate variables can be your friend, helping to break down the structure (similar to functions in modular programming).

EDIT the second: When I first looked at your problem, I thought the lowercase $h$ variables were vectors. Now I realize they are matrices, which changes things. Given this, it does not make sense to me that the result would be a matrix, rather than a higher-order tensor. The order of a tensor is the number of indices. If you take a derivative of one tensor $U$ with respect to another tensor $V$, the order of the resulting tensor $W$ will be the sum of the two, i.e. $$W_{ij,pq}=\frac{\partial U_{ij}}{\partial V_{pq}}$$

For instance the gradient of a scalar function is a vector ($0+1=1$), and the Jacobian of a vector function is a matrix ($1+1=2$). For an example of a physically-meaningful $4_{th}$ order tensor, a simple example is the "spring constant" for the continuum-mechanics generalization of Hooke's Law (the last equation on that page is essentially the same as the one I give above).

When I work through your case, I find that the result is sparse, so is effectively $3_{rd}$ order, but it is not so sparse as to be $2_{nd}$ order $\ldots$ unless I made a mistake. (More precisely: I find $W_{ij,pq}=0$ for $p\neq i$, so we can effectively work with $\hat{W}_{ij,k}\equiv W_{ij,ik}$ as an order-3 tensor.)

EDIT the third: As the OP requested clarification, here is a simple example to demonstrate the issue. Consider the matrix equation $$U=VA\implies U_{ij}=\sum_kV_{ik}A_{kj}\implies \frac{\partial U_{ij}}{\partial V_{pq}}=\sum_k\frac{\partial V_{ik}}{\partial V_{pq}}A_{kj}$$ To proceed, we introduce the Kronecker delta symbol (essentially the identity matrix): $$\delta_{ij} = \begin{cases} 1 & i=j\\ 0 & i\neq j \end{cases}$$ Then the derivative within the summation can be expressed as $$\frac{\partial V_{ik}}{\partial V_{pq}}=\delta_{ip}\delta_{kq}$$ i.e. it is one if the both indices match and zero otherwise.

Noting the "index replacement" property of a summed tensor-$\delta$ product $$\sum_k\delta_{kq}A_{kj}=A_{qj}$$ we then have $$\frac{\partial U_{ij}}{\partial V_{pq}}=\delta_{ip}A_{qj}$$

So finally $$W_{ij,pq}= \begin{cases} A_{qj} & p=i\\ 0 & p\neq i \end{cases}$$ (and $\hat{W}_{ij,k}=A_{kj}$)

• Thank you for your help. I took a week to cogitate on your thorough and thoughtful response and realized my gap in understanding was in tensor calculus. I still don't know why I had to transpose that Whh matrix but I guess after going more material ill try to revisit it Sep 14, 2016 at 8:47
• @Moose I have no idea if I am doing these correctly or just confusing myself! Anyway, I answered a similar question here, for another example. Sep 18, 2016 at 1:36