I have coded a very basic LSTM with forget gates (no libraries used). I'm trying to predict 0.5*sin(t + N) given 0.5*sin(t) as an exercise.

I have tweaked the model, changing the output layer activation function, weight initialization, number of memory blocks/cells, etc. However, I still couldn't manage to correct the output.

The problem is that the output range is much smaller than desired, [-0.2, 0.2] instead of [-0.5, 0.5]. The output also is slightly delayed, meaning it is predicting sin(t + N - 1) for example.

Is there something that I'm missing?

As an example, for output layer activation function as a centered logistic from (-1, 1), the validation output looks like

Validation Output

Training output looks like

Training Output

Additional information:

Topology: 1 input layer, 1 hidden layer each with 5 memory blocks each with 1 cell, 1 output layer each with 1 regular neuron.

Alpha: 1

Weights: generated with normal distribution, from [-1, 1]

Output layer activation function used: logistic [0, 1], centered logistic, tanh, ReLU, leaky ReLU, f(x) = x (identity)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.