# Why is LSTM so ineffective at this ridiculously simple sequence?

I am trying to predict the the sequence $$(0.1, 0.05, …, 1.1)$$ from the sequence $$(0, 0.05, 0.1, …, 1)$$. I thought this would be the only item in a toy dataset.

Now I implemented a long-short term memory LSTM model in PyTorch and I can't figure out why it trains so badly.

With learning rate $$10^{-3}$$ and $$10^3$$ iterations on this single data point it fails to converge effectively. Results are bad.

With learning rate $$10^{-4}$$ and $$10^4$$ loss decreases more, and results are slightly better

Here is the code:

import torch import numpy as np

lstm = torch.nn.LSTM(input_size=1, hidden_size=100, num_layers=1, batch_first=True).to(torch.double)
linear = torch.nn.Linear(100,1).to(torch.double)
criterion = torch.nn.MSELoss()
params = list(lstm.parameters()) + list(linear.parameters())

L = 21

x = torch.tensor(np.arange(L)/20, dtype=torch.double).unsqueeze(0).unsqueeze(2)
y = torch.tensor(np.arange(2,L+2)/20,  dtype=torch.double)

for j in range(1000):
hidden = (torch.zeros((1,1,100), dtype=torch.double), torch.zeros((1,1,100), dtype=torch.double))
x = x
y = y
output, hidden = lstm(x, hidden)
loss = 0.0
for i in range(L):
o = linear(output[:,i,:].squeeze())
loss += criterion(o, y.squeeze()[i])
print(loss)
loss.backward()
optimizer.step()

out_seq = torch.empty((L)) for i in range(L):
o = linear(output[:,i,:].squeeze())
out_seq[i] = o
print("output (expected): ", y) print("output:", out_seq)


It seems unreasonable to me that it requires to spend so much time on a single data sequence to figure out it has just to add 0.1 to each element. So I'd appreciate any insight on how to improve that.

• I think the target sequence is supposed to be $(0.1,0.15,…,1.1)$ instead of $(0.1,0.05,…,1.1)$. Nov 14, 2018 at 7:55
• it doesn't even look like a sequential problem. How does knowledge about previous element in the sequence will effect prediction for the next element? Each element of the sequence can be trained independently from one another Nov 15, 2018 at 10:43

optimizer.zero_grad()  # add this line

I added this fix in your code and I get progress during the training. I got loss equal to 0.0005 after 1000 iterations with step 1e-3.