I came across this answer which is specific to Keras. But my question is at concept level. I am getting confused, How can RNN handle variable size inputs? The answer here says RNN can handle variable sequence inputs.

My question is, How exactly the weights matrix W (see my figure) work with different length inputs?

Let us suppose we want to do a sentiment classification.

Training batches of different length : Which I think means all training examples in a batch must be of same length but different batches can have variable length training examples.

Now, suppose length of training examples in

  • batch-1 = 10,
  • batch-2 = 15.
  • batch-3 = 18 ... so on

Now while training batch-1 , In the unrolled RNN there would be 10 columns in the below figure to handle input of sequence 10. By columns I mean different timestamps in RNN. The shared weights W, U,V dimensions must be in sync with sequence length. What I mean by this is, the shared weights to handle sequence of length 10 would be something like w1,w2,..w10 (horizontal connection that connects columns.) I understand till this part.

Now, while processing batch-2 which is of sequence length 15, The unrolled RNN must take sequence of length 15. That means there are 15 columns in the below figure and corresponding weights will be like w1,w2,..w15.

The number of weights W changed from 10(w1,w2,...w10) to 15(w1,w2,..w15) as we go from batch-1 to batch-2.

The W dimension changes. Things don't add up here for me.

What happens to shared weights? How are they learnt?

enter image description here


The point of the RNN is that, in your terminology, w1...w10 are actually all the same sets of weights. The weights are not different for different time steps. So really you have w1=w2=w3=...=w10=w. There are only one set of weights for an RNN regardless of how long the input sequence is. Conceptually you are feeding your outputs back into the inputs of the same neuron - hence recurrence. You therefore only need to store one W which is reused at every time step.


Not the answer you're looking for? Browse other questions tagged or ask your own question.