RNN weight and state matrices

Question

1. Implementations of RNN in NLP tasks, like those in https://dennybritz.com/posts/wildml/recurrent-neural-networks-tutorial-part-2/, are done using matrices, that are used to store the inputs, outputs, memory(state) and weights of the RNN. Since a matrix has a predefined size, I understand that this size limits the variable size to a maximum sentence lenght that the RNN can deal with. Is it correct?

2. Assuming that W is the weight matrix applied to the neurons of the hidden-to-hidden layer, why is W defined as a square matrix of dimensions "HxH", with "H" being the number of hidden layers (or the maximum number of hidden layers)? Why isn´t it only a vector of "H" size with one weight for each hidden layer?

Thanks!

Answer 1

1. The matrix dimensions are unrelated to the maximum length of the input. This becomes clear if you examine the equations for whatever RNN variety you're interested in: they all have recurrence in common, so the same weights are re-used to predict $t+1$ using data at $t$ and hidden state information $h_t$.
2. The hidden weight dimension $H \times H$ refers to the number of units in the hidden layers: $H$ inputs mapped to $H$ outputs.