3
$\begingroup$

In my opinion there are basically 4 weight matrix in the RNN. SO there are different names given in different scenarios but let me point out what I know about RNN in very simple terms. Please correct if wrong. Let us look at this image. enter image description here

  1. H: State matrix. Which is the one responsible for the decision at any point of time. It's the only one that changes and records the Memory in the network. With each input at state time t, it'll change and get passed on to the next cell. In the end, it is basically used for prediction in a classification task.
  2. Wx matrix or the Input Weight Matrix which will get multiplied to the Input at each time step. It is common for every time step.
  3. Wh or the Hidden State Matrix which get multiplied to the H and changes the state. ( obviously after using the input multiplication and activation function). It is also same for each time step.
  4. Wy or the Output Matrix which is responsible to give an output at each time step for Multi-out model like Named Entity Recognition after getting multiplied by the State Matrix: H. It is also shared.

I want to know if my understanding of RNN is right or wrong.

Also, if I have to explain it to a new person, can I say that these tokens/ Inputs are passed in Sequential manner because the diagram is just an illustration and parallelism is not possible because the results are dependent on the previous step and the State has to change before you pass in a new input?

Improve this question
$\endgroup$
1
  • $\begingroup$ @AryaMcCarthy Thanks for helping man. Yeah I might have forgotten it. And I also read the paper somewhere else too. Got the idea. Thanks again. $\endgroup$
    – Deshwal
    Commented Apr 3, 2021 at 16:10

1 Answer 1

Reset to default
4
$\begingroup$

Everything in your understanding is correct, except for a nomenclature thing.

I would caution you that H isn’t actually a weight matrix. The words “weights” and “parameters” are pretty interchangeable. In the case of H, it’s not a parameter. It depends on the particular sequence x that you observe.

It’s not entirely true that you can’t parallelize the RNN computation over the sequence lengths see Martin and Cundy (2018) for how the prefix sum algorithm can be used to build up the computation for several segments of the input sequence, then merge these. (It’s similar to how carry-lookahead adders improve on ripple-carry adders.) This only works for certain classes of RNNs.

In general, though, it’s fair to say what you said. The sequential dependencies in the RNN make parallelization impossible.

Improve this answer
$\endgroup$
3
  • $\begingroup$ So what is H exactly? It must be a vector, right? This is where we do LSTM(20) so H is a vector of length 20? Even though it is not learned directly but is dependent on other learned matrices and input, but still it is exactly what we want to get updated the most. Right? $\endgroup$
    – Deshwal
    Commented Apr 3, 2021 at 16:12
  • $\begingroup$ Depends on how you define $H$. I’ll stick to one way: It’s a vector of length 20, yup. But it’s not a parameter of the model. It’s the model state as you process the input. It depends on both the weights of the model and the input you’ve read so far. I don’t know what you mean by “it is exactly what we want to get updated the most”. $\endgroup$
    – Arya McCarthy
    Commented Apr 3, 2021 at 16:22
  • $\begingroup$ I meant that it it the only thing which we return while classification. We return Hidden state in the end so that we can classify based on this state? $\endgroup$
    – Deshwal
    Commented Apr 4, 2021 at 7:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.