I would like to seek some clarifications on the dimensionalities of the components and weight parameters in a vanilla RNN model performing text classification for the next word. I will present my understanding which is unclear towards the end. Please point out any part that is wrong.

If each word is vectorised over a vocabulary of size $K$, then the input_size = number of features is $K$. I have a input $X$ of size (batch_size ($n_x$), seq_len ($n$), input_size($K$)). If the hidden layer has dimension $M$, then the input-to-hidden weight matrix $W_{hx}$ has size ($M, n_x$). The hidden-to-hidden weight matrix has size ($M, M$).

At step $t$, $x_t$ has size ($n_x, K$). \begin{align} a_t&= W_{hx}x_t + W_{hh}h_{t-1} + b_h \\ h_t &= \text{tanh}(a_t) \end{align} So both the activation $a_t$ and the hidden state $ h_t$ have size ($M, K$).

I find it not very clear from this point on. If the hidden-to-output weight matrix $W_{yh}$ has size ($K,M$), \begin{align} \hat{y}_t &= \text{softmax}(W_{yh}h_t) \end{align} the predicted output at step $t$, $\hat{y}_t$ by my intuition will have size ($K,1$), because each entry is supposed to represent the probability of the $k$-th word in the vocabulary appearing next. But here it seems to become ($K,K$).

Please help me with the dimensionalities. Thanks in advance.


Your input-to-hidden matrix $W_{hx}$ has shape $M \times K$. Your hidden matrix $W_{hh}$ has shape $M \times M$. Then $h_t, b_h, a_t$ all have shape $M$. The output matrix $W_{yh}$ has shape $K \times M$, so $W_{yh}h_t$ has shape $K$. Softmax doesn't change any shapes, so your output is $K$.

You seemed confused about whether to think about $x_t, h_t$, etc as a single input/state or an entire batch of them. It doesn't really matter, as long as you're consistent about it. Above, I wrote it out as if $x_t, h_t$, etc are a single input. In batch form, we would have $x_t$ is $B \times K$, and $h_t, a_t$ are $B \times M$. Then $\hat y_t$ is $B \times K$.

Of course in batch form, the notation for matrix multiplication is a bit more convoluted -- it doesn't make sense to write $W_{hx} x_t$ anymore, you have to write this as $x_t W_{hx}^T$ which now has the correct shape $B \times M$.

If you don't want to think about time index $t$ either, you could write it out as: $X$ is $B \times N \times K$, and $H, A$ are both $B \times N \times M$, and $\hat Y$ is $B \times N \times K$.

  • $\begingroup$ Thanks for your answer! Could you help me with the further question in the edited post? $\endgroup$
    – siegfried
    Nov 28 '21 at 4:08
  • $\begingroup$ @siegfried If you have an additional question, you should ask it using the ASK QUESTION button at the top of the page. You can link back to this question if you think it provides helpful context $\endgroup$
    – Sycorax
    Nov 28 '21 at 4:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.