Take the first step $t=1$ for simplicity, \begin{align} h_1 &= \text{tanh}(W_{hx}^T e_1 + W_{hh}^T h_0) \end{align} Usually it is written in convention: \begin{align} \frac{\partial L_1}{\partial W_{hx}} &= \frac{\partial h_1}{\partial W_{hx}} \frac{\partial L_1}{\partial h_1} \end{align} where $e_1$ is the first-step input of shape $(n_i,1)$, $W_{hx}$ is the input-to-hidden matrix of shape $(n_i, n_h)$, $W_{hh}$ the hidden-to-hidden matrix of shape $(n_h, n_h)$.
$L_1$ is the first-step cross-entropy scalar, $h_1$ is the hidden state of shape $(n_h,1)$.

It can also be expressed as \begin{align} \frac{\partial L_1}{\partial W_{hx}} &= \sum_{i=1}^{n_h} \frac{\partial h_1^i}{\partial W_{hx}} \frac{\partial L_1}{\partial h_1^i} \end{align} where $h_1^i$ is the $i$-th unit in vector $h_1$. On the RHS each term is a $(n_i, n_h)$ matrix multiplied by a scalar $\frac{\partial L_1}{\partial h_1^i}$.

$\frac{\partial L_1}{\partial W_{hx}} $ should have the same shape as the weight matrix, thus $(n_i, n_h)$.

$\frac{\partial L_1}{\partial h_1}$ as I understand has the same shape as $h_1$, thus $(n_h,1)$.

In particular the dimension of $\frac{\partial h_1}{\partial W_{hx}}$ is confusing. The term $\frac{\partial h_1^i}{\partial W_{hx}} $ seems to be an element in the Kronecker product $\frac{\partial h_1}{\partial W_{hx}} $. I would like some help sorting out this term.

My attempt:

Let $f= W_{hx}^T e_1$. \begin{align} f &= I_{n_h} W_{hx}^T e_1 \\ &= (e_1^T \otimes I_{n_h}) \text{vec}(W_{hx}^T) \\ &= (e_1^T \otimes I_{n_h}) w \\ \frac{\partial f}{\partial w} &= e_1^T \otimes I_{n_h} \end{align}

Then \begin{align} \frac{\partial h_1}{\partial w} &= (e_1^T \otimes I_{n_h}) \text{ diag}( 1- h_1^2) \end{align} where $a_1 = W_{hx}^T e_1 + W_{hh}^T h_0 $.

But the problems with this are:

  1. This is $\frac{\partial h_1}{\partial \text{ vec}(W_{hx})}$, not $\frac{\partial h_1}{\partial W_{hx}}$.

  2. How do I understand $e_1^T \otimes I_{n_h}$?

  • $\begingroup$ Your last expression is correct $\endgroup$
    – J. Delaney
    Commented Feb 25, 2022 at 17:36
  • $\begingroup$ In your edit, why did you introduce $h_1^T$ in the chain rule, instead of $h_1$? $\endgroup$
    – gunes
    Commented Mar 5, 2022 at 13:15

1 Answer 1


The first multiplicand in the chain rule is a vector derivative by a matrix, which is a 3D tensor, and its product is not well defined. This situation is usually resolved by flattening the matrices and reformulating the equations using Kronecker products as you did in the second part of your post, or the derivatives are expressed in an element-wise fashion, then combined together.

From your equations, it can be deduced that you've chosen denominator layout. In this layout, the following derivative is $$\frac{\partial (Ax)}{\partial x}=A^T$$ so, it's not $A$, (see here), which means

$$ \begin{align} \frac{\partial h_1}{\partial w} &= (e_1^T \otimes I_{n_h})^T \text{diag}( 1- h_1^2)\\&= (e_1 \otimes I_{n_h})\text{diag}( 1- h_1^2) \end{align} $$

This equation is consistent with the inner dimensions, unlike the version in your post. When multiplied with ${{\partial L}\over{\partial h_1}}$, this corresponds to the same formula as the summation you've included, just a reshaped version.

If matrix equations is chosen, the derivative should be reshaped back into its original form (i.e. de-vec operation).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.