5
$\begingroup$

I am trying to see the power of recurrent neural calculations. I give the NN just one feature, a timeseries datum one step in the past, and predict a current datum. The timeseries is however double-seasonal with considerably long ACF structure (about 64) with additive shorter seasonality for lag 6.

Input timeseries: enter image description here

Validation result: enter image description here

You could note it is shifted. I checked my vectors, and they seem OK.

MSE residuals are also quite bad (I expect 0.01 on both train validation thanks to Gaussian noise added with sigma = 0.1): enter image description here

> head(x_train)
[1]  0.9172955  0.9285578  0.4046166 -0.4144658 -0.3121450  0.3958689
> head(y_train)
           [,1]
[1,]  0.9285578
[2,]  0.4046166
[3,] -0.4144658
[4,] -0.3121450
[5,]  0.3958689
[6,]  1.5823631

Q: am I doing something wrong in terms of LSTM acrchitecture, or data preparation, or batching?

library(keras)
library(data.table)

# constants

features <- 1
timesteps <- 1

x_diff <- sin(seq(0.1, 100, 0.1)) + sin(seq(1, 1000, 1)) + rnorm(1000, 0, 0.1)

#x_diff <- ((x_diff - min(x_diff)) / (max(x_diff) - min(x_diff)) - 0.5) * 2


# generate  training data

train_list <- list()
train_y_list <- list()

for(
     i in 1:(length(x_diff) / 2 - timesteps)
    )
{
     train_list[[i]] <- x_diff[i:(timesteps + i - 1)]
     train_y_list[[i]] <- x_diff[timesteps + i]
}

x_train <- unlist(train_list)
y_train <- unlist(train_y_list)

x_train <- array(x_train, dim = c(length(train_list), timesteps, features))
y_train <- matrix(y_train, ncol = 1)


# generate  validation data

val_list <- list()
val_y_list <- list()

for(
     i in (length(x_diff) / 2):(length(x_diff) - timesteps)
)
{
     val_list[[i - length(x_diff) / 2 + 1]] <- x_diff[i:(timesteps + i - 1)]
     val_y_list[[i - length(x_diff) / 2 + 1]] <- x_diff[timesteps + i]
}

x_val <- unlist(val_list)
y_val <- unlist(val_y_list)

x_val <- array(x_val, dim = c(length(val_list), timesteps, features))
y_val <- matrix(y_val, ncol = 1)


## lstm (stacked) ----------------------------------------------------------

# define and compile model
# expected input data shape: (batch_size, timesteps, features)

rm(fx_model)

fx_model <- 
     keras_model_sequential() %>% 
     layer_lstm(
          units = 32
          #, return_sequences = TRUE
          , input_shape = c(timesteps, features)
          ) %>% 
     #layer_lstm(units = 16, return_sequences = TRUE) %>% 
     #layer_lstm(units = 16) %>% # return a single vector dimension 16
     #layer_dropout(rate = 0.5) %>% 
     layer_dense(units = 4, activation = 'tanh') %>% 
     layer_dense(units = 1, activation = 'linear') %>% 
     compile(
          loss = 'mse',
          optimizer = 'RMSprop',
          metrics = c('mse')
     )


# train

# early_stopping <-
#      callback_early_stopping(
#           monitor = 'val_loss'
#           , patience = 10
#           )

history <- 
     fx_model %>% 
     fit( 
     x_train, y_train, batch_size = 50, epochs = 100, validation_data = list(x_val, y_val)
)

plot(history)

## plot predict

fx_predict <- data.table(
     forecast = as.numeric(predict(
          fx_model
          , x_val
     ))
     , fact = as.numeric(y_val[, 1])
     , timestep = 1:length(x_diff[(length(x_diff) / 2):(length(x_diff) - timesteps)])
)

fx_predict_melt <- melt(fx_predict
                        , id.vars = 'timestep'
                        , measure.vars = c('fact', 'forecast')
                        )

ggplot(
     fx_predict_melt[timestep < 301, ]
       , aes(x = timestep
             , y = value
             , group = variable
             , color = variable)
       ) +
     geom_line(
          alpha = 0.95
          , size = 1
     ) +
     ggplot_theme
$\endgroup$
9
  • 1
    $\begingroup$ How big is your data? Presumably you would be better of with an autoregressive modell which allows for double seasonalities, e.g. double seasonal Holt-Winters oder tbats. I would use a tbats model. $\endgroup$ – Ferdi Apr 6 '18 at 11:37
  • 1
    $\begingroup$ @Ferdi, thank you. I can actually use TBATS (or, simpler, lm) for a benchmark against LSTM. My goal, however, is to understand limits and opportunities of the later. My data are synthetic, as you could see in the code attached. All the hype around LSTM networks should somehow be justified by being able to get information from timeseries's deeper lags, as I get it, at least. $\endgroup$ – Alexey Burnakov Apr 6 '18 at 12:14
  • 3
    $\begingroup$ LSTM is rather for time-series with many observations per time-point. Furthermore with a small or medium-sized you might be terribly overfitting with an LSTM or an RNN. $\endgroup$ – Ferdi Apr 6 '18 at 12:17
  • 1
    $\begingroup$ @Ferdi, I tried to add 64 lags into feature vector (which should be enough to capture both seasonal patterns with reasonably low residuals). It did much better now. But... I don't see the point of using the recurrent part, since the same result is easy to obtain using a regular fully connected NN. $\endgroup$ – Alexey Burnakov Apr 6 '18 at 15:06
  • 3
    $\begingroup$ I'm constantly amazed by the number of "LSTM for forecasting time series" blog posts and kaggle kernels which then show a forecast which is essentially a one step ahead naive forecast. Like you @AlexeyBurnakov I am mystified by the hype LSTM seem to be getting for forecasting. $\endgroup$ – Skander H. Apr 10 '18 at 3:27
2
$\begingroup$

So, after trying many input and parameter tweaks, I came to a conclusion that LSTM cannot long dependencies until it gets long enough vector of past time series values. In my experiments a so-so good quality of forecast could be obtained after feeding the net with 64 lags, which span over the seasonalities in the model.

Another thing is that minibatches are a bad idea if they were sampled randomly. In the realization of neural networks I played with I made it work with 100% of examples passed in iteration. That way I ensured that all examples come in time-wise sequences.

Also it is worth mentioning that the LSTM result compared poorly against a linear benchmarking model.

If you think I am wrong, give me good counter arguments.

$\endgroup$
3
  • 1
    $\begingroup$ Lstm can manage long dependencies. I wrote a tutorial to deal with time series with Keras ahstat.github.io/RNN-Keras-time-series does it helps you? See also my answer on SO stackoverflow.com/questions/41947039/… Presented code is in Python and for series outputs, but you may adapt it $\endgroup$ – ahstat Apr 10 '18 at 9:14
  • $\begingroup$ @ahstat, I understand the maximum depth of dependency in your timeseries is 3, like this: "y3(t)=x4(t−3) for t≥3"? I am working with a dependency going back for 64 steps: y(t) = x(t - 64), in your notations. My experiment was whether LSTM can learn from so long pattern using only x(t-1) as input. $\endgroup$ – Alexey Burnakov Apr 10 '18 at 9:59
  • $\begingroup$ Yes, see post of Philippe Remy here philipperemy.github.io/keras-stateful-lstm for an example with very long dependencies $\endgroup$ – ahstat Apr 10 '18 at 10:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.