2
$\begingroup$

When working with an LSTM network in Keras. The first layer has the input_shape parameter show below.

model.add(LSTM(50, input_shape=(window_size, num_features), return_sequences=True))

I don't quite follow the window size parameter and the effect it will have on the model. As far as I understand, to make a decision the network not only makes use of current windowframe but also the information about past windows stored in the network.

So is the window size more of a tool for saving memory / computing requirements? Or does it have a big impact on a model?

$\endgroup$

1 Answer 1

1
$\begingroup$

So is the window size more of a tool for saving memory / computing requirements? Or does it have a big impact on a model?

It's both! Imagine you have a long text like War and Peace. Back-propagating from the end of the text to the beginning is a huge effort because the text is so long. Most of the effect of the update will pertain to the most recent time-steps, because the previous words in a sentence are most relevant for what you're predicting (the next word). On the other hand, imagine the most extreme truncation, which only looks back 1 time step. This won't allow the model to learn any long-term dependencies because the model focuses exclusively on the most recent time-step.

Picking a good window size is important, but fine-tuning (e.g. choosing between 64 and 65) isn't necessary -- pick a window that's "large enough" to learn longer dependencies, and call it a day.

The term of art for truncating the number of time steps in a recurrent neural network is "truncated back-propagation through time."

$\endgroup$
4
  • $\begingroup$ thanks for your answer. I am still a little confused, apologies I am a slow learner. As I understand it the batch size dictates how many observations are grouped together for back propagating. I also read that you don't take advantage of the long term memory any more or less by changing batch size so are we saying that it is the window size that specifies how long the model should look back to find a relationship? $\endgroup$
    – mHelpMe
    May 13, 2020 at 20:01
  • $\begingroup$ But I when I read the answer to another of my questions stats.stackexchange.com/questions/385911/… it appears to me that nothing is even thrown out of the model so that through back propagating the long term memory shows which i think is linked to the batch size $\endgroup$
    – mHelpMe
    May 13, 2020 at 20:02
  • 1
    $\begingroup$ I don't understand your question. Batch size and window size are two different concepts. $\endgroup$
    – Sycorax
    May 13, 2020 at 21:00
  • $\begingroup$ Apologies I keep confusing myself. I thought batch size was basically the window size and didn't really understand why the window size required because the batch size dictates when the model updated the weights in the model. Think I need to start again and understand the difference between Batch size & window size $\endgroup$
    – mHelpMe
    May 14, 2020 at 17:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.