For research purposes, I want to predict one time series by data of the other. I use data from GDELT and attached you may find a plot of the course of both series. Time series

Remarkably, two sharp drops appear. One around 2012 and the other around 2014. Since both drops appeared in both series simultaneously at the same time, it raised suspicion. Turns out that values are calculated differently from 2012 until 2014 and from 2014 until 2016. It is the same feature, however, calculations from 2005 until 2012 differ from 2012 until 2014 which differ from 2014 until 2016. I want to predict the period of 2014 until 2016 and train/validate on the data of 2005 until 2014. These drops are problematic. I am not sure what transformation to use such that, these drops will be taken care of and I can train models.

I know about subtracting the mean of the period from the values within that period. I am sure there are multiple ways of doing a transformation. I am not confident which one to apply and how to justify.

Hopefully, light can be shed on this issue.

  • $\begingroup$ If the values were calculated differently in the different time periods, I do not believe that a transformation is a good way to solve your problem. Your idea of subtracting the mean would work only if you can be confident that the mean remained the same across these time periods, which seems unlikely. $\endgroup$ – mkt - Reinstate Monica Nov 14 '17 at 17:49
  • $\begingroup$ Can you think of a way that transforms the data such that 2005-2014 can be used for training and 2014-2016 can be used for testing purposes? $\endgroup$ – Stan Nov 14 '17 at 18:18
  • $\begingroup$ Given the clear regime structure to the measurement changes, one approach would be to use dummy (0,1) variables to account for them. Personally, I can't think of any transformations that would work with this data and, if one were found, give you any confidence that it wouldn't result in significant retransformation bias. There are lots of ways to validate the model, i.e., no one is forcing you to use the last 2 years for this purpose. K-fold cross validation and bootstrapping into train and test would also be valid approaches. $\endgroup$ – Mike Hunter Nov 14 '17 at 18:52

I would normalize the data in each interval by subtracting the mean over that interval and dividing by the standard deviation over that interval.

Rather than subtracting the mean alone, which will leave the variance of the data in those later years on a different scale, normalizing will put all variability on the same scale, and that's what really matters to a machine learner.

However, if the way sentiment was measured in these periods really did dramatically change rather than just get put on a slightly different scale, I would not expect your model to give good predictions. Because your data dips from positive to negative, it suggests this may be the case.

  • $\begingroup$ @gung I tried to delete the comment but I couldn't // didn't know how ! $\endgroup$ – IrishStat Nov 14 '17 at 19:57
  • $\begingroup$ I guess i inadvertently left the answer ... $\endgroup$ – IrishStat Nov 14 '17 at 19:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.