I've recently become interested in LSTMs and I was surprised to learn that the weights are shared across time.

  • I know that if you share the weights across time, then your input time sequences can be a variable length.

  • With shared weights you have many fewer parameters to train.

From my understanding, the reason one would turn to an LSTM vs. some other learning method is because you believe there is some sort of temporal/sequential structure/dependence in your data that you would like to learn. If you sacrifice the variable length ‘luxury’, and accept long computation time, wouldn’t an RNN/LSTM without shared weights (i.e. for every time step you have different weights) perform way better or is there something I’m missing?

up vote 6 down vote accepted

The 'shared weights' perspective comes from thinking about RNNs as feedforward networks unrolled across time. If the weights were different at each moment in time, this would just be a feedforward network. But, I suppose another way to think about it would be as an RNN whose weights are a time-varying function (and that could let you keep the ability to process variable length sequences).

If you did this, the number of parameters would grow linearly with the number of time steps. That would be a big explosion of parameters for sequences of any appreciable length. It would indeed make the network more powerful, if you had the massive computational resources to run it and the massive data to constrain it. For long sequences, it would probably be computationally infeasible and you'd get overfitting. In fact, people usually go in the opposite direction by running truncated backpropagation through time, which only unrolls the network for some short period of time, rather than over the entire sequence. This is done for computational feasibility. Interestingly, RNNs can still learn temporal structure that extends beyond the truncation length, because the recurrent units can store memory from before.

  • If you don't share weights, you still have the cell state that persists across time. An unrolled LSTM with unique time weights would look like a feedforward net where each 'layer' would represent a time slice, but each 'layer' would have incoming cell state information. It would resemble a feedforward, but with the addition of cell state. – beeCwright Feb 22 '17 at 17:51

The accepted answer focuses on the practical side of the question: it would require a lot of resources, if there parameters are not shared. However, the decision to share parameters in an RNN has been made when any serious computation was a problem (1980s according to wiki), so I believe it wasn't the main argument (though still valid).

There are pure theoretical reasons for parameter sharing:

  • It helps in applying the model to examples of different lengths. While reading a sequence, if RNN model uses different parameters for each step during training, it won't generalize to unseen sequences of different lengths.

  • Oftentimes, the sequences operate according to the same rules across the sequence. For instance, in NLP:

                                                     "On Monday it was snowing"

                                                     "It was snowing on Monday"

...these two sentences mean the same thing, though the details are in different parts of the sequence. Parameter sharing reflects the fact that we are performing the same task at each step, as a result, we don't have to relearn the rules at each point in the sentence.

LSTM is no different in this sense, hence it uses shared parameters as well.

I think since the RNNs with hidden-to-hidden recurrences (and time shared weights) are equivalent to Universal Turing Machines, letting them have different weights for different time steps does not make them more powerful.

  • Could you elaborate on what you mean by "powerful"? The reference to Turing Machines suggests what you have in mind may be completely different than what is meant in statistics. – whuber Feb 4 '17 at 17:27
  • The RNNs are used to process sequence of data. One of their commonest types gets a sequence as input and produces another sequence as output (such as language translation systems). I say that an RNN model family M1 is more powerful than another RNN model family M2, if for a problem (such as mapping a set of input sequences to a set of output sequences) there is some model m1 in M1 where can solve this problem but there is no model in M2 where can solve that problem. – Hossein Feb 4 '17 at 20:21

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.