# LSTM - Learning a sinus function with linear part

I have recently build a simple LSTM-Network to predict a sinus function, which worked fine. Now I wanted to fit a sinus function containing a linear part with the same network but the results are disappointing. The code below generates the data, trains the network and prints the resulting graph and the training history so you can easily reproduce it yourself. The code below makes 20 predictions with a window size of 50 and prints it in the graph. I also scaled the data to a range of [0,1], since LSTM-Networks tend to have problems on non scaled data.

My questions in particular are.

1. Why does the LSTM network has problems when a linear part is added to the sinus function?
2. How can the model be improved, so that it fits the data.
3. Why does the training history looks "good", or at least not obviously bad.

I already tried reducing the batch size, making the network wider/deeper

This is the resulting prediction and as you can see it doesn't fit the true data well.

As far as I understand machine learning the learning curve looks not like under/overfitting, but I am not an expert.

import matplotlib.pyplot as plt
import numpy as np
from keras.models import Sequential
from sklearn.preprocessing import MinMaxScaler
from keras.layers import Dense, LSTM

def build_model(window_size):
model = Sequential()
return model

def create_dataset(x, y, window_size, train_split, scaler):
y = y.reshape(y.shape[0], 1)
dataset = scaler.fit_transform(y)
print(dataset)

# create sliding window
window_size = window_size + 1
windows = []
for i in range(len(dataset) - window_size):
windows.append(dataset[i: i + window_size])

windows = np.array(windows)

# split in train/test
rows = round(train_split * windows.shape[0])
train = windows[:int(rows), :]
x_train = train[:, :-1]
y_train = train[:, -1]
x_test = windows[int(rows):, :-1]
y_test = windows[int(rows):, -1]

# convert in 3d-tensor for LSTM network
x_train = np.reshape(x_train, (x_train.shape[0], x_train.shape[1], 1))
x_test = np.reshape(x_test, (x_test.shape[0], x_test.shape[1], 1))

return [x_train, y_train, x_test, y_test]

def plot_results_multiple(predicted_data, true_data, prediction_len):
fig = plt.figure(facecolor='white')
ax.plot(true_data, label='True Data')
# Pad the list of predictions to shift it in the graph to it's correct start
for i, data in enumerate(predicted_data):
padding = [None for p in range(i * prediction_len)]
# print(type(data))
plt.legend()
plt.show()

def predict_sequences_multiple(model, data, window_size, prediction_len):
prediction_seqs = []
for i in range(int(len(data) / prediction_len)):
curr_frame = data[i * prediction_len]
predicted = []
for j in range(prediction_len):
predicted.append(model.predict(curr_frame[np.newaxis, :, :])[0, 0])
curr_frame = curr_frame[1:]
curr_frame = np.insert(curr_frame, [window_size - 1], predicted[-1], axis=0)
prediction_seqs.append(predicted)
return np.array(prediction_seqs)

# generating the sin function with a linear part
x = np.arange(0, 100, 0.1)
a = 0.4 * x + 3
y = np.sin(x) + a

scaler = MinMaxScaler(feature_range=(0,1))
window_size = 50
n_predictions = 20

model = build_model(50)
model.compile(loss='mse', optimizer='rmsprop', metrics=['mae'])
x_train, y_train, x_test, y_test = create_dataset(x, y, window_size, 0.7, scaler)
history = model.fit(x_train, y_train, validation_data=(x_test, y_test), batch_size=512, epochs=50, verbose=2, shuffle=False)
predictions = predict_sequences_multiple(model, x_test, window_size, n_predictions)

# inverse transform to oroginal values
predictions = scaler.inverse_transform(predictions)
y_test = scaler.inverse_transform(y_test)
plot_results_multiple(predictions, y_test, n_predictions)

plt.plot(history.history['loss'][:])
plt.plot(history.history['val_loss'][:])
plt.title('train and validation loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper right')
plt.show()

• How many cycles you have in your training sample? – Aksakal Feb 15 '18 at 16:52
• I generated the data with x = np.arange(0, 100, 0.1). The above graph shows 4 cycles an is only the 30% testset. Overall it has 12 cycles to train and 4 to test. – Dennis Feb 15 '18 at 17:00
• NN are good at interpolation, they are not so good at extrapolation. try fitting your network to a trend line, without sine wave, just a plain line and see how it fares – Aksakal Feb 15 '18 at 17:23
• It seems that Aksakal is right but I thought scaling the data would solve this issue. However the questions remain, since they are not answered yet and I 'd like to understand the reason. – Dennis Feb 15 '18 at 17:54
• If you want to build NN that predicts anything, any kind of time series then it's a very ambitious goal. In traditional time series analysis we spend considerable amount of time understanding the properties of series, whether they have trends, seasonaility etc., then we pick appropriate methods to model them. So, I'd suggest that if you know that the series have a trend then incorporate this knowledge in the NN, particularly into features. Your series are probably $x_t=at+sin(t)$, so take care of this $at$ terms somehow, maybe include $x_t-x_{t-lag}$ as a feature or time itself and so on. – Aksakal Feb 15 '18 at 18:09