I am going through the following blog on LSTM neural network: http://machinelearningmastery.com/understanding-stateful-lstm-recurrent-neural-networks-python-keras/

The author reshapes the input vector X as [samples, time steps, features] for different configuration of LSTMs.

The author writes

Indeed, the sequences of letters are time steps of one feature rather than one time step of separate features. We have given more context to the network, but not more sequence as it expected

What does this mean?


I found this just below the [samples, time_steps, features] you are concerned with.

X = numpy.reshape(dataX, (len(dataX), seq_length, 1))

Samples - This is the len(dataX), or the amount of data points you have.

Time steps - This is equivalent to the amount of time steps you run your recurrent neural network. If you want your network to have memory of 60 characters, this number should be 60.

Features - this is the amount of features in every time step. If you are processing pictures, this is the amount of pixels. In this case you seem to have 1 feature per time step.

  • 1
    $\begingroup$ can you explain the difference between : X = numpy.reshape(dataX, (len(dataX), 3, 1)) and X = numpy.reshape(dataX, (len(dataX), 1, 3)) How does this affect the lstm? $\endgroup$ – Vipul Jain Feb 28 '17 at 13:39
  • 1
    $\begingroup$ (len(dataX), 3, 1) runs LSTM for 3 iterations, inputting a input vector of shape (1,). (len(dataX), 1, 3) runs LSTM for 1 iteration. Which means that it is quite useless to even have recurrent connections since there can't be any feedback from previous iterations. In this case input shape to RNN is of shape (3,) $\endgroup$ – Joonatan Samuel Feb 28 '17 at 14:24
  • $\begingroup$ "(len(dataX), 3, 1) runs LSTM for 3 iterations" dont we use epoch for that. does it means same as epoch=3? $\endgroup$ – Vipul Jain Feb 28 '17 at 16:02
  • 1
    $\begingroup$ One epoch in rough translation means that we have trained once on every data point in our data set. While training, having looked through len(dataX) examples counts as 1 epoch. However, RNNs take data in sequentially. On every training example, you have to feed it data over multiple iterations. E.g, I have a word "car" and on every iteration I feed it one letter, let it complete computation and then feed the next letter. In order to complete processing the word "car" it needs 3 iterations to process the whole word letter by letter. $\endgroup$ – Joonatan Samuel Feb 28 '17 at 16:25
  • $\begingroup$ @JoonatanSamuel hi, I know a lot of time has passed but I am having a lot of trouble understanding the same topic. Your answer is very clear, but I am still a bit confused. Immagine we had a time series describing sales at each month for several years (say). Imagine len(data) = 3000, data.shape = (3000,1) for instance, so we have 3000 months recorded. Predict the next: If I want to use N previous observation to predict the next one (only the next one!), what should the shape of the input data to LSTM be? For instance if we wanted to use t-n,..., t-2, t-1 to predict t. $\endgroup$ – Euler_Salter Oct 13 '17 at 9:28

It's a bit too late but just in case;
A Sample may refer to individual training examples. A “batch_size” variable is hence the count of samples you sent to the neural network. That is, how many different examples you feed at once to the neural network.

TimeSteps are ticks of time. It is how long in time each of your samples is. For example, a sample can contain 128-time steps, where each time steps could be a 30th of a second for signal processing. In Natural Language Processing (NLP), a time step may be associated with a character, a word, or a sentence, depending on the setup.

Features are simply the number of dimensions we feed at each time steps. For example in NLP, a word could be represented by 300 features using word2vec. In the case of signal processing, let’s pretend that your signal is 3D. That is, you have an X, a Y and a Z signal, such as an accelerometer’s measurements on each axis. This means you would have 3 features sent at each time step for each sample.

By Guillaume


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.