# LSTM NN produces “shifted” forecast (low quality result)

I am trying to see the power of recurrent neural calculations. I give the NN just one feature, a timeseries datum one step in the past, and predict a current datum. The timeseries is however double-seasonal with considerably long ACF structure (about 64) with additive shorter seasonality for lag 6.

Input timeseries:

Validation result:

You could note it is shifted. I checked my vectors, and they seem OK.

MSE residuals are also quite bad (I expect 0.01 on both train validation thanks to Gaussian noise added with sigma = 0.1):

> head(x_train)
[1]  0.9172955  0.9285578  0.4046166 -0.4144658 -0.3121450  0.3958689
[,1]
[1,]  0.9285578
[2,]  0.4046166
[3,] -0.4144658
[4,] -0.3121450
[5,]  0.3958689
[6,]  1.5823631


Q: am I doing something wrong in terms of LSTM acrchitecture, or data preparation, or batching?

library(keras)
library(data.table)

# constants

features <- 1
timesteps <- 1

x_diff <- sin(seq(0.1, 100, 0.1)) + sin(seq(1, 1000, 1)) + rnorm(1000, 0, 0.1)

#x_diff <- ((x_diff - min(x_diff)) / (max(x_diff) - min(x_diff)) - 0.5) * 2

# generate  training data

train_list <- list()
train_y_list <- list()

for(
i in 1:(length(x_diff) / 2 - timesteps)
)
{
train_list[[i]] <- x_diff[i:(timesteps + i - 1)]
train_y_list[[i]] <- x_diff[timesteps + i]
}

x_train <- unlist(train_list)
y_train <- unlist(train_y_list)

x_train <- array(x_train, dim = c(length(train_list), timesteps, features))
y_train <- matrix(y_train, ncol = 1)

# generate  validation data

val_list <- list()
val_y_list <- list()

for(
i in (length(x_diff) / 2):(length(x_diff) - timesteps)
)
{
val_list[[i - length(x_diff) / 2 + 1]] <- x_diff[i:(timesteps + i - 1)]
val_y_list[[i - length(x_diff) / 2 + 1]] <- x_diff[timesteps + i]
}

x_val <- unlist(val_list)
y_val <- unlist(val_y_list)

x_val <- array(x_val, dim = c(length(val_list), timesteps, features))
y_val <- matrix(y_val, ncol = 1)

## lstm (stacked) ----------------------------------------------------------

# define and compile model
# expected input data shape: (batch_size, timesteps, features)

rm(fx_model)

fx_model <-
keras_model_sequential() %>%
layer_lstm(
units = 32
#, return_sequences = TRUE
, input_shape = c(timesteps, features)
) %>%
#layer_lstm(units = 16, return_sequences = TRUE) %>%
#layer_lstm(units = 16) %>% # return a single vector dimension 16
#layer_dropout(rate = 0.5) %>%
layer_dense(units = 4, activation = 'tanh') %>%
layer_dense(units = 1, activation = 'linear') %>%
compile(
loss = 'mse',
optimizer = 'RMSprop',
metrics = c('mse')
)

# train

# early_stopping <-
#      callback_early_stopping(
#           monitor = 'val_loss'
#           , patience = 10
#           )

history <-
fx_model %>%
fit(
x_train, y_train, batch_size = 50, epochs = 100, validation_data = list(x_val, y_val)
)

plot(history)

## plot predict

fx_predict <- data.table(
forecast = as.numeric(predict(
fx_model
, x_val
))
, fact = as.numeric(y_val[, 1])
, timestep = 1:length(x_diff[(length(x_diff) / 2):(length(x_diff) - timesteps)])
)

fx_predict_melt <- melt(fx_predict
, id.vars = 'timestep'
, measure.vars = c('fact', 'forecast')
)

ggplot(
fx_predict_melt[timestep < 301, ]
, aes(x = timestep
, y = value
, group = variable
, color = variable)
) +
geom_line(
alpha = 0.95
, size = 1
) +
ggplot_theme

• How big is your data? Presumably you would be better of with an autoregressive modell which allows for double seasonalities, e.g. double seasonal Holt-Winters oder tbats. I would use a tbats model. – Ferdi Apr 6 '18 at 11:37
• @Ferdi, thank you. I can actually use TBATS (or, simpler, lm) for a benchmark against LSTM. My goal, however, is to understand limits and opportunities of the later. My data are synthetic, as you could see in the code attached. All the hype around LSTM networks should somehow be justified by being able to get information from timeseries's deeper lags, as I get it, at least. – Alexey Burnakov Apr 6 '18 at 12:14
• LSTM is rather for time-series with many observations per time-point. Furthermore with a small or medium-sized you might be terribly overfitting with an LSTM or an RNN. – Ferdi Apr 6 '18 at 12:17
• @Ferdi, I tried to add 64 lags into feature vector (which should be enough to capture both seasonal patterns with reasonably low residuals). It did much better now. But... I don't see the point of using the recurrent part, since the same result is easy to obtain using a regular fully connected NN. – Alexey Burnakov Apr 6 '18 at 15:06
• I'm constantly amazed by the number of "LSTM for forecasting time series" blog posts and kaggle kernels which then show a forecast which is essentially a one step ahead naive forecast. Like you @AlexeyBurnakov I am mystified by the hype LSTM seem to be getting for forecasting. – Skander H. Apr 10 '18 at 3:27