28
$\begingroup$

I am going through the following blog on LSTM neural network: http://machinelearningmastery.com/understanding-stateful-lstm-recurrent-neural-networks-python-keras/

The author reshapes the input vector X as [samples, time steps, features] for different configuration of LSTMs.

The author writes

Indeed, the sequences of letters are time steps of one feature rather than one time step of separate features. We have given more context to the network, but not more sequence as it expected

What does this mean?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
25
$\begingroup$

I found this just below the [samples, time_steps, features] you are concerned with. 

X = numpy.reshape(dataX, (len(dataX), seq_length, 1))

Samples - This is the len(dataX), or the amount of data points you have.

Time steps - This is equivalent to the amount of time steps you run your recurrent neural network. If you want your network to have memory of 60 characters, this number should be 60.

Features - this is the amount of features in every time step. If you are processing pictures, this is the amount of pixels. In this case you seem to have 1 feature per time step.

Improve this answer
$\endgroup$
11
  • 1
    $\begingroup$ can you explain the difference between : X = numpy.reshape(dataX, (len(dataX), 3, 1)) and X = numpy.reshape(dataX, (len(dataX), 1, 3)) How does this affect the lstm? $\endgroup$
    – Vipul Jain
    Commented Feb 28, 2017 at 13:39
  • 1
    $\begingroup$ (len(dataX), 3, 1) runs LSTM for 3 iterations, inputting a input vector of shape (1,). (len(dataX), 1, 3) runs LSTM for 1 iteration. Which means that it is quite useless to even have recurrent connections since there can't be any feedback from previous iterations. In this case input shape to RNN is of shape (3,) $\endgroup$
    – Joonatan Samuel
    Commented Feb 28, 2017 at 14:24
  • $\begingroup$ "(len(dataX), 3, 1) runs LSTM for 3 iterations" dont we use epoch for that. does it means same as epoch=3? $\endgroup$
    – Vipul Jain
    Commented Feb 28, 2017 at 16:02
  • 1
    $\begingroup$ One epoch in rough translation means that we have trained once on every data point in our data set. While training, having looked through len(dataX) examples counts as 1 epoch. However, RNNs take data in sequentially. On every training example, you have to feed it data over multiple iterations. E.g, I have a word "car" and on every iteration I feed it one letter, let it complete computation and then feed the next letter. In order to complete processing the word "car" it needs 3 iterations to process the whole word letter by letter. $\endgroup$
    – Joonatan Samuel
    Commented Feb 28, 2017 at 16:25
  • $\begingroup$ @JoonatanSamuel hi, I know a lot of time has passed but I am having a lot of trouble understanding the same topic. Your answer is very clear, but I am still a bit confused. Immagine we had a time series describing sales at each month for several years (say). Imagine len(data) = 3000, data.shape = (3000,1) for instance, so we have 3000 months recorded. Predict the next: If I want to use N previous observation to predict the next one (only the next one!), what should the shape of the input data to LSTM be? For instance if we wanted to use t-n,..., t-2, t-1 to predict t. $\endgroup$
    – Euler_Salter
    Commented Oct 13, 2017 at 9:28
7
$\begingroup$

My answer with an example: ["hello this is xyz","how are you doing","great man..."]

in this case "[samples, time steps, features]" means:

  • sample: 3 because there are 3 elements in the list
  • time steps: here you can take max_length = 4 length("hello this is xyz") = 4; length("how are you doing") = 4; length("great man...") = 2 (after removing punctuation "."). The reason of saying this is a time steps is, in first element "hello this is xyz" ==> t0("hello"), t1("this"), t2("is") and t3("xyz")
  • features: the size of embedding for each words. e.g, "hello": 50D array, "this": 50D array and so on
Improve this answer
$\endgroup$
2
  • $\begingroup$ How would you construct your input if you wanted to take into account the raw words as well as their Part of Speech tagging (POS)? I.e. in my project I think that the word sequences as well as the POS sequences would be good features for the model. Any recommendation on how to merge them? E.g. words-sequence followed by POS-sequence, or w1 pos1 w2 pos2 w3 pos3... ? $\endgroup$
    – Alaa M.
    Commented Sep 5, 2021 at 11:52
  • $\begingroup$ Following your example, if we perform text classification to predict the next word, what will be the shape of the RNN output $\hat{y}$ be? (n_sample, n_steps, then what??) $\endgroup$
    – siegfried
    Commented Nov 27, 2021 at 1:36
5
$\begingroup$

It's a bit too late but just in case;
A Sample may refer to individual training examples. A “batch_size” variable is hence the count of samples you sent to the neural network. That is, how many different examples you feed at once to the neural network.

TimeSteps are ticks of time. It is how long in time each of your samples is. For example, a sample can contain 128-time steps, where each time steps could be a 30th of a second for signal processing. In Natural Language Processing (NLP), a time step may be associated with a character, a word, or a sentence, depending on the setup.

Features are simply the number of dimensions we feed at each time steps. For example in NLP, a word could be represented by 300 features using word2vec. In the case of signal processing, let’s pretend that your signal is 3D. That is, you have an X, a Y and a Z signal, such as an accelerometer’s measurements on each axis. This means you would have 3 features sent at each time step for each sample.

By Guillaume

Improve this answer
$\endgroup$
1
  • $\begingroup$ Since you mentioned NLP, how would you construct your input if you wanted to take into account the raw words as well as their Part of Speech tagging (POS)? I.e. in my project I think that the words sequence as well as the POS sequence would be good features for the model. Any recommendation how to merge them? E.g. words-sequence followed by POS-sequence, or w1 pos1 w2 pos2 w3 pos3... ? $\endgroup$
    – Alaa M.
    Commented Sep 5, 2021 at 11:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.