LSTM clarification on output

Question

I (think), I understand how LSTMs are roughly working. I need some clarifications though.

1. Since there can be an output at each time step, the "output dimension" (in the sense of many to one or many to many, I don't know how to name it besides "output dimension") can't be bigger than the number of time steps, is that right?

2. If the number of time steps $d$ is bigger than my "output dimension" $k$, do I always use the last $k$ time steps as my outputs, or rather something like every (total number of time steps)/$k$ time steps one output? And how can I adjust that in Keras?

3. Is there any heuristic on how many outputs I should use (something like, the longer the time dependencies are, the more outputs or something similar)

Answer 1

• output dimension and number of time steps are entirely orthogonal
• each time you send in one input into the network => you get out one output. Each time you send one input in, and get one output, this is one timestep
• you can send in as many inputs as you want, one at a time, in sequence, and you'll get the same number of outputs (for a typical LSTM, though alternative implementations are possible)
• the dimensionality of each input is not connected with the number of timesteps
• nor is the dimensionality of each output connected to the number of timesteps

As far as number of timesteps, so... during prediction time, you can keep feeding in inputs, and getting outputs. Typically the output is a prediction for the next input, though it doesnt have to be this way.

For training... you have to backprop through all your timesteps. But because the gradients vanish after 10-50 timesteps or so, what we normally do is use truncated back-propagation through time, BPTT. What this means is:

• you take a set of timesteps, say 50, or 30, or whatever seems 'about right' for you
• feed them forwards through the network, generating predictions, which you store
• then back propagate, through the same timesteps, in reverse order, generating gradients as you go, and using the predicted outputs from forwards direction, to generate your error signal
• and thats it :-)
• pick another sets of inputs, and repeat

Edit: in the light of your question about many-to-one, I think you might be thinking about sequence-to-sequence and similar. Basically, three ways of using an RNN, related to this concept are:

• one timestep in => get one prediction out (this is what I discussed above). You might use this to predict the next word of a sentence for example
• multiple timesteps in => one vector out. What you do is, pass each of the timesteps through, and then take the hidden state of the RNN as the output. In this way you get multiple timesteps in, one vector out, many to one
• you can also do sequence to sequence, which is two RNNs back to back (could be the same RNN, and/or shared weights:
  • pass the input timesteps into the first RNN, get the hidden state of this RNN as an intermediate output vector (many to one)
  • initialize the hidden state of the second RNN with this intermediate output vector, and run the second RNN, until you hit some kind of 'end of sequence' token (one to many)
  • overall, this sequence to sequence architecture then gives many to many