0
$\begingroup$

I want to predict datasets like this

enter image description here

I'm using Keras' LSTM:

enter image description here

Not like a classification problem which returns probabilities, this model needs to output the exact value depending on input giving to it. It means that the output might be a very large positive/negative number.

Q1. About activation

I noticed that the output of my model was very small, I think the reason is that the default LSTM activation is Tanh which output in range(-1,1), so I changed the activation from Tanh to linear. Besides, the model predictions were poor when using "Tanh" activation on LSTM in this case.

Am I doing this right? Or there's some other best activation I could use?

Q2. Do I need to Standardize the data?

Do I need to standardize input? But if I scale the input how can the model return the exact value?

Q3. Single target and shift target

single target

  • dataset: [1,2,3,4,5,6]
  • input: [1,2,3],[2,3,4],[3,4,5]
  • target:[ 4 ],[ 5 ],[ 6 ]

enter image description here

shift target

  • dataset: [1,2,3,4,5,6]
  • input: [1,2,3],[2,3,4],[3,4,5]
  • target:[2,3,4],[3,4,5],[4,5,6]

enter image description here

In my case, I got better results on using single target, but some of the examples are using the shifting target like this tutorial.

What is the difference between using the shifting target and the single target?

$\endgroup$
1
$\begingroup$

Q1. About activation

I would stick with the tahn. After all that is why you have the dense layer at the end. The activation function of the dense layer should be linear simply to do a projection of the output of the lstm cell in the desired range.

Q2. Do I need to Standardize the data?

Two options I think:

  1. Standardize the input not the output. The correct outputs are the true numbers, but the inputs are the normalized ones. Then, when you perform inference you would need to directly use the outputs of the networks as inputs, so before providing the input just normalize it.
  2. Normalize both, and then do inverse transform with the normalizer you are using on the output to get the true labels. However, this should be a tougher problem for the network since the difference between the outputs will be smaller.

Q3. Single target and shift target

The single target is kind of like the standard approach. The only problem being is that the gradients need to be propagated from the end of the network all the way to the start in order to adjust the weighs accordingly. Now, if you use a shift target, you are measuring the performance of the network at each time step, hence shorter gradient propagation. Usually it improves performance (read a couple of papers where they stated that it did, but for your case it might be different).

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

Q1. About activation

Yes. You can either use "linear" as LSTM activation or add a Dense layer with no activation as you final layer. (With no activation, Dense would be just a linear transformation, like "linear").

Q2. Do I need to Standardize the data?

Technically, you don't have to scale the input, as the linear transformation and bias at the first layer should take care of it. In experience though, I have achieved better results when normalized the input data. (Possibly due to initialization of biases to zero...)

One option is to first split the data into trn, vld, and tst. Then, calculate the mean and std of the trn set and normalize trn, vld, and tst data. Then, make your input and output signals per trn, vld, and tst and train the network. When you want to make predictions out there, first normalize the data with the same mean and std, and multiply the output by std and add the mean.

Q3.

Well, this depends how much in the future you want to make predictions, just a single time-step or multiple. If it is single, I would use the network under "single target". The second target and the output shapes do not make much sense to me. Typically you should have only one dimension as None which would be the number of samples.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.