2
$\begingroup$

I'm new to time-series analysis using neural networks (but not to neural networks in general and programming).

I have a 1D time-series of 47249 observations and a fairly slow computer, so training is slow. My idea is to somehow split the time-series into batches and update model parameters for each batch, as opposed to feeding the whole series into the recurrent network and updating parameters based on the entire output. That way the model can "learn" from each batch and apply its "knowledge" to the next batches, instead of being terribly wrong on the whole sequence.

The problem is, how do I split the time-series into batches and how do I manage the hidden state between batches?

I saw several approaches online:

  • This notebook on Google Colab uses sliding windows (over 1D data) as input, so one batch looks like this:

    [[0.0154, 0.0270, 0.0541, 0.0483],
     [0.0270, 0.0541, 0.0483, 0.0328],
     [0.0541, 0.0483, 0.0328, 0.0598], ...]
    

    The code to run the model (PyTorch) looks something like this (this function is run every time you apply your model to data):

    def forward(self, x):
        batch_size = x.size(0)
        # Hidden states for LSTM
        h_0 = torch.zeros(self.num_layers, batch_size, self.hidden_size)
        c_0 = torch.zeros(self.num_layers, batch_size, self.hidden_size)
    
        # Propagate input through LSTM
        ula, (h_out, _) = self.lstm(x, (h_0.requires_grad_(), c_0.requires_grad_()))
    
        # ...
    
        return ...
    

    However, the entire point of using a recurrent network is that it works on sequences without the need for windows, so why introduce them here? Moreover, after the model processes the first window, [0.0154, 0.0270, 0.0541, 0.0483], its next input from the second window will be 0.0270, but in the original time-series the next input is 0.0328 - the last element of the second window. The order of elements looks incorrect to me...

    Moreover, the forward function sets the hidden state to zero for each new input, so it treats each batch (or maybe each window in the batch??) as a separate entity, yet these windows are parts of one sequence, and the model's "memory" (hidden states) should reflect the fact that, for example, the second batch is the continuation of the first and the zeroth batches. Doesn't this approach reduce the model's memory?

  • This article also sets the hidden states to zero for each batch, which, IMO, also cuts the model's memory. Moreover, it shuffles the batches, which destroys the order of the batches. However, isn't order absolutely crucial for time-series? Thus, this code also treats each batch as a separate sequence, even though it contains subsequences of the original sequence.

  • This article also resets the hidden states for each batch (or subsequence?).

The point here is that these articles reset the hidden states for each batch, even though these batches are part of the same sequence, so the hidden state should be maintained between batches. Despite this, the articles show that these models perform fairly well.

I got the idea of maintaining the hidden state between batches from Flux.jl:

... it is important to note that a single continuous sequence is considered. Since the model state is not reset between the 2 batches, the state of the model flows through the batches, which only makes sense in the context where seq_1 is the continuation of seq_init and so on.


I attempted to share the hidden state between batches like this in PyTorch:

class MyLSTM(nn.Module):
    def __init__(self, in_size: int):
        super().__init__()
        
        # LSTM
        self.hidden_size = 10
        self.lstm = nn.LSTM(
            input_size=in_size, hidden_size=self.hidden_size,
            batch_first=True
        )

        # Fully-connected layers
        self.seq = nn.Sequential(
            nn.Linear(self.hidden_size, 5), nn.Tanh(),
            nn.Linear(5, 1), nn.Tanh()
        )
        self.reset_hidden()
        
    def reset_hidden(self):
        "Requests reset of hidden state"
        self.h0 = self.c0 = None
        
    def forward(self, x):
        batch_size = x.size(0)
        
        if self.h0 is None:
            # Resets hidden state ONLY if reset was requested!!
            self.h0 = torch.zeros(1, batch_size, self.hidden_size, requires_grad=True)
            self.c0 = torch.zeros(1, batch_size, self.hidden_size, requires_grad=True)
        
        out, (self.h0, self.c0) = self.lstm(x, (self.h0[:, :batch_size, ...], self.c0[:, :batch_size, ...]))
        
        return self.seq(out)

This code will reset the hidden states self.h0, self.c0 only when a reset is requested.

The training loop requests a reset when a new epoch begins, because that's when the time series will be fed to the model again from the beginning. Otherwise the code is standard:

for epoch in range(100):
    model.reset_hidden()

    for X, y in data_loader:
        y_hat = model(X)
        loss = loss_function(y_hat, y)
        
        optimizer.zero_grad()
        loss.backward()
        optimizer.zero_grad.step()

This, however, causes an error that talks about autograd being unable to backprop because parts of the computation graph have been destroyed:

RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.

This kind of makes sense because this code:

  1. back propagates the gradients through the hidden states for the first batch,
  2. destroys intermediate values in the computation graph afterwards,
  3. updates the hidden states when processing the second batch,
  4. attempts to backprop through the same hidden states for the second batch, but now it needs to differentiate the hidden states with respect to themselves in the previous batch, but that information was lost in step 2!

Calling loss.backward(retain_graph=True) also produces an error:

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [10, 40]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

Enabling anomaly detection as suggested by the error message shows that the offending line is:

out, (self.h0, self.c0) = self.lstm(x, (self.h0[:, :batch_size, ...], self.c0[:, :batch_size, ...]))

...which makes sense, because the entire point is to update self.h0 and self.c0!

Questions

  • What is the proper way of splitting the time-series into batches? Should I use a sliding window? Or non-overlapping windows?
  • How do I manage LSTM's hidden state between batches? Should I just reset it for each batch? Wouldn't this defeat the purpose of using a recurrent network?

EDIT: looks like my question is similar to the following:

  • LSTM: how to feed the Network with a mini batch? When to reset the LSTM state?
    • The answer there says to reset the state "for each sample", but it's unclear what a sample is? Is a batch a sample? Is each window inside the batch a sample?
  • Understanding how to batch and feed data into a stateful LSTM
    • The question itself talks about resetting the state for each batch, like I'm trying to do, but the answer doesn't mention the LSTM state at all.
    • Also, people in the comments are recommending "overlapping samples with stateless LSTM", which is what the first Colab notebook is doing. However, it's still unclear how that would impact the model's memory. Surely one can do better than destroying the state after each batch?
$\endgroup$
4
  • $\begingroup$ There is some information in this link discuss.pytorch.org/t/… $\endgroup$ Jul 8, 2022 at 16:56
  • 1
    $\begingroup$ "I have a 1D time-series of 47249 observations" -- does this mean you have 47,249 different 1D time-series? Or something else? It would be helpful if you can explain what your data is, because the answer depends crucially on what its structure is. For example, you might have 47,249 randomly-selected Tweets, in which case tracking data from tweet to tweet is unimportant. Or you might have some structure, such as 37 different weather stations that measured the temperature at 1277 different time steps. So the answer is it depends; please edit to clarify. $\endgroup$
    – Sycorax
    Jul 8, 2022 at 17:04
  • $\begingroup$ This Question might be helpful: stats.stackexchange.com/questions/395382/… $\endgroup$
    – Sycorax
    Jul 8, 2022 at 17:14
  • $\begingroup$ @Sycorax, no, it was one time-series with 47k numbers $\endgroup$
    – ForceBru
    Jul 10, 2022 at 15:16

0