I have coded a very basic LSTM with forget gates (no libraries used). I'm trying to predict 0.5*sin(t + N) given 0.5*sin(t) as an exercise.
I have tweaked the model, changing the output layer activation function, weight initialization, number of memory blocks/cells, etc. However, I still couldn't manage to correct the output.
The problem is that the output range is much smaller than desired, [-0.2, 0.2] instead of [-0.5, 0.5]. The output also is slightly delayed, meaning it is predicting sin(t + N - 1) for example.
Is there something that I'm missing?
As an example, for output layer activation function as a centered logistic from (-1, 1), the validation output looks like
Training output looks like
Topology: 1 input layer, 1 hidden layer each with 5 memory blocks each with 1 cell, 1 output layer each with 1 regular neuron.
Weights: generated with normal distribution, from [-1, 1]
Output layer activation function used: logistic [0, 1], centered logistic, tanh, ReLU, leaky ReLU, f(x) = x (identity)