Omitting the bias term, the recursion looks like:
$$h_{t+1}=tanh(x_{t} U+W h_{t}) =tanh(p), \textrm{say}$$ where the tanh is taken elementwise.
Hence, $$\frac{\partial h_{i,t+1} }{\partial h_{j,t}} =(1-tan^2{p})\frac{\partial p }{\partial h_{t}}$$
The diagonal structure might come from the cross derivatives being zero depending on whether the layers are fully connected or not . I'll work it out and post details