1
$\begingroup$

Assuming Keras LSTM will reset internal state after each batch, I want to understand how internal state is maintained within a single batch. Suppose then, batch_size=4, timesteps=3 and num_features=1. The problem is to predict the next element of a univariate timeseries $x_1,x_2, ... ,x_n$.

According to guides, the right way to feed this model is to prepare overlapping windows of length timesteps. Thus, the first batch consists of the following inputs and targets:

input A: [$x_1,x_2,x_3$] $\rightarrow$ target A: $x_4$

input B: [$x_2,x_3,x_4$] $\rightarrow$ target B: $x_5$

input C: [$x_3,x_4,x_5$] $\rightarrow$ target C: $x_6$

input D: [$x_4,x_5,x_6$] $\rightarrow$ target D: $x_7$

Because this is a single batch, the internal state should be maintained throughout. First, the model will process input A using the initial internal state (the state after reset). The new internal state after processing input A will be the internal state produced after processing $(x_1,x_2,x_3)$, which I will denote as $h_{3}$. Similarly, the internal states after processing $(x_1)$ and $(x_1,x_2)$ are $h_{1}$ and $h_{2}$, however these are not maintained.

Next, the model will process input B. However, if I understood correctly, it will process input B given an internal state $h_3$. Thus, it will process $(x_2,x_3,x_4)$ given than it has previously seen $(x_1,x_2,x_3)$. Yet, how could it be that the model has seen $x_3$, as it about to be fed with $x_2$, followed by $x_3$ and $x_4$? In contrary, it makes more sense to me that it should process input B given an internal state $h_{1}$, not $h_{3}$.

In summary, it seems as if the sequence of internal state is based on a sequence $(x_1,x_2,x_3,x_2,x_3,x_4,x_3,x_4,x_5...)$ and not the original sequence $(x_1,x_2,x_3,x_4,...)$. How is this not wrong?

Also, if the network has already been fed with $x_3$ in input A , how can it be fed again two more times with $x_3$ in input B and input C, without messing up the internal state?

$\endgroup$
0

1 Answer 1

0
$\begingroup$

If ABCD are given as 4 different observations, then the state will not be consistent between them. If you give one sequence $x_1, x_2, ..., x_t$ and do autoregressive prediction (predict $x_{i+1}$ given $i$ for 1,2,...), the state will be consistent (because there's only 1 input). An easy way to verify this is to write a simple test case comparing the two scenarios.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.