1
$\begingroup$

I have a time series data set for actual number of airport passengers. Within 15 years (2004 ~ 2019), just like having a trend, number of the passengers is increasing over time as the country is becoming more and more popular, which is good for my country. (more recent time a little steeper increase in the passengers then further past years)

But when I now try to do time series analysis for future passenger prediction, it is not a good event I presume. Because, I did standardization (mean and SD) for training data set (2004 ~ 2015) then applied the coefficient to validation set (2016~ 2018) and test set (2019). The problem is, number of passengers were much smaller in training data set time (2004 - 2015), but during validation and test set data time (2016~2019), quite strong increase in number of passengers is shown. That being said, standardization on training data set worked nicely (mostly around 0 ~ 1), but in validation and test set, it is about 7 ~ 9 as the standardization coefficient is coming from far behind past (training set) which is small. I know data leakage is a problem so I should not violate but I think because of this big difference in the standardization result, my LSTM models are not working properly.

So my fundamental question is,

How can I solve training/validation/test set with normalization/standardization under this situation? should I still stick to the fitting only to the training set and transforming validation and test set with it? or should there be a different approach?

I would really be glad if there are productive feedback.

Thank you very much in advance.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Don't standardize on the entire dataset (that's leaking data, and I think it's cheating).

It seems like what you're experiencing is due to the trend in your data. One way to solve this is to predict the first difference (change in the number of passengers). Assuming the first difference is not trending, this should solve the problem. This way, your later data points can be standardized with the same standardization as the training set.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for your comment. I tried difference actually beforehand. But it did not really help in practice. and LSTM is supposed to be largely free from the effect of trend unlike in ARIMA. So eliminating trend will really help? $\endgroup$
    – S. Jay
    Nov 21, 2019 at 1:40
  • 1
    $\begingroup$ Well your issue isn't with the LSTM, it's with the standardization. When you take the first difference, does that time series still have a trend? If so, you may need to take the second difference. The other thing you can do is include a time variable in your model so the model can capture the trend explicitly. $\endgroup$
    – David Atlas
    Nov 21, 2019 at 13:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.