I am trying to predict the the sequence $(0.1, 0.05, …, 1.1)$ from the sequence $(0, 0.05, 0.1, …, 1)$. I thought this would be the only item in a toy dataset.

Now I implemented a long-short term memory LSTM model in PyTorch and I can't figure out why it trains so badly.

With learning rate $10^{-3}$ and $10^3$ iterations on this single data point it fails to converge effectively. Results are bad.

With learning rate $10^{-4}$ and $10^4$ loss decreases more, and results are slightly better

Here is the code:

import torch import numpy as np

lstm = torch.nn.LSTM(input_size=1, hidden_size=100, num_layers=1, batch_first=True).to(torch.double)
linear = torch.nn.Linear(100,1).to(torch.double)
criterion = torch.nn.MSELoss()
params = list(lstm.parameters()) + list(linear.parameters())
optimizer = torch.optim.Adam(params, 1e-3)

L = 21

x = torch.tensor(np.arange(L)/20, dtype=torch.double).unsqueeze(0).unsqueeze(2)
y = torch.tensor(np.arange(2,L+2)/20,  dtype=torch.double)

for j in range(1000):
    hidden = (torch.zeros((1,1,100), dtype=torch.double), torch.zeros((1,1,100), dtype=torch.double))
    x = x
    y = y
    output, hidden = lstm(x, hidden)
    loss = 0.0
    for i in range(L):
        o = linear(output[:,i,:].squeeze())
        loss += criterion(o, y.squeeze()[i])

out_seq = torch.empty((L)) for i in range(L):
    o = linear(output[:,i,:].squeeze())
    out_seq[i] = o
     print("output (expected): ", y) print("output:", out_seq)

It seems unreasonable to me that it requires to spend so much time on a single data sequence to figure out it has just to add 0.1 to each element. So I'd appreciate any insight on how to improve that.

  • 1
    $\begingroup$ I think the target sequence is supposed to be $(0.1,0.15,…,1.1)$ instead of $(0.1,0.05,…,1.1)$. $\endgroup$
    – Brandmaier
    Nov 14, 2018 at 7:55
  • $\begingroup$ it doesn't even look like a sequential problem. How does knowledge about previous element in the sequence will effect prediction for the next element? Each element of the sequence can be trained independently from one another $\endgroup$
    – itdxer
    Nov 15, 2018 at 10:43

2 Answers 2


You have bug in the code. It looks like torch accumulates gradients and before making backpropagation you need to clean them (set to zero)

optimizer.zero_grad()  # add this line

I added this fix in your code and I get progress during the training. I got loss equal to 0.0005 after 1000 iterations with step 1e-3.

Check this answer from the stackoverflow. You can also check this RNN tutorial on github


In principle, the network should learn to entirely by-pass the LSTM cell(s), ie., close their output gate(s) forever. Adding 0.1 to the input needs only a bias weight (that is, a constant) added to the input. So, you don't even need a hidden layer at all. Some of the earlier LSTM work improved results dramatically by choosing the "right" bias weights for the LSTM gates. After all, you have a large network with many degrees-of-freedom and only few data points, so it may be tricky for the network to converge to that solution from your starting values.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.