Why are the weights of RNN/LSTM networks shared across time?

Question

I've recently become interested in LSTMs and I was surprised to learn that the weights are shared across time.

• I know that if you share the weights across time, then your input time sequences can be a variable length.

• With shared weights you have many fewer parameters to train.

From my understanding, the reason one would turn to an LSTM vs. some other learning method is because you believe there is some sort of temporal/sequential structure/dependence in your data that you would like to learn. If you sacrifice the variable length ‘luxury’, and accept long computation time, wouldn’t an RNN/LSTM without shared weights (i.e. for every time step you have different weights) perform way better or is there something I’m missing?

Answer 1

The accepted answer focuses on the practical side of the question: it would require a lot of resources, if there parameters are not shared. However, the decision to share parameters in an RNN has been made when any serious computation was a problem (1980s according to wiki), so I believe it wasn't the main argument (though still valid).

There are pure theoretical reasons for parameter sharing:

• It helps in applying the model to examples of different lengths. While reading a sequence, if RNN model uses different parameters for each step during training, it won't generalize to unseen sequences of different lengths.

• Oftentimes, the sequences operate according to the same rules across the sequence. For instance, in NLP:

                                                     "On Monday it was snowing"

                                                     "It was snowing on Monday"

...these two sentences mean the same thing, though the details are in different parts of the sequence. Parameter sharing reflects the fact that we are performing the same task at each step, as a result, we don't have to relearn the rules at each point in the sentence.

LSTM is no different in this sense, hence it uses shared parameters as well.

Answer 2

The 'shared weights' perspective comes from thinking about RNNs as feedforward networks unrolled across time. If the weights were different at each moment in time, this would just be a feedforward network. But, I suppose another way to think about it would be as an RNN whose weights are a time-varying function (and that could let you keep the ability to process variable length sequences).

If you did this, the number of parameters would grow linearly with the number of time steps. That would be a big explosion of parameters for sequences of any appreciable length. It would indeed make the network more powerful, if you had the massive computational resources to run it and the massive data to constrain it. For long sequences, it would probably be computationally infeasible and you'd get overfitting. In fact, people usually go in the opposite direction by running truncated backpropagation through time, which only unrolls the network for some short period of time, rather than over the entire sequence. This is done for computational feasibility. Interestingly, RNNs can still learn temporal structure that extends beyond the truncation length, because the recurrent units can store memory from before.

Answer 3

I think since the RNNs with hidden-to-hidden recurrences (and time shared weights) are equivalent to Universal Turing Machines, letting them have different weights for different time steps does not make them more powerful.

Answer 4

I am trying hard to visualize how weight sharing combined with recurrence and combined with word embeddings behaves in a high-dimensional space.

Taking the example from @Maxim and visualizing a network that suggests the next word in the sequence: "On Monday it was" when accumulated using recurrence will be a point in a high dimensional space, and thanks to word embeddings, "On Tuesday it was" will be in the same manifold. Given this accumulated weight as an input to a downstream fully connected layer with high memory capacity, it will learn to map to things like cold, snowing, etc. There may be other stored mappings like hectic, slow, obvious, etc. This may be learnt by one unit of the layer. Another unit may have learnt to map the high dimensional vector formed from the accumulated weight of "It was snowing on" to vectors like christmas, Monday, the, etc. This is about hidden-hidden weights and hidden-output weights. About input-hidden weights, although the weights are shared, the units of the layer that they lead to will be activated for different aspects of a sentence (people and places, stop words, etc), making them position (time) agnostic.

Answer 5

RNN is a time based neural network.. at the end of time steps ( length of the input) it forms a vector which represents a thought preserving sequence information across the time. Thinking of thought vector like some sort of figure or object might help, which gets it's proper shape (depending on the input sequence) through time steps depending on the inputs it see's each time. The weight matrices are initialized randomly first, If we take example as predicting the next letter using RNN, when we send the first letter and the network predicts the next letter by assigning probabilities to each possible letters. we can update the weights using the gradients in that timestep. same goes for all the letters until the word ends. At the end the weights gets updated in such a way that, it would increase the confidence ( probability ) of finding the right word which can be achieved through backpropagation. This Training process continues for large number of data, thus tuning the weight parameters in such a way that given the sequence seen so far and this is the current state then this particular letter/ word ( in case of machine translations ) has the high probability of occurring. Weight Sharing across the time stamps thus helps in understanding the sequence as well as in applied point of view reduces the training time.