1
$\begingroup$

I have ~1500 time series data representing store sales (US$). All time series are of the same size with 52 weeks of data with no NA values. For a subset of 18 specific time series, I want to find the top 10 "control" time series that match based on similarity metrics.

The similarity matching process is:
1) Pearson Correlation between time series (negative correlations are filtered out)
2) Engle-Granger cointegration (two-step process as defined in link)
3) Euclidean Distance (the smaller the distance, the more similar the time series are)

Note that I normalize the data before applying the above process, defined as subtracting the mean and dividing by SD for each individual time series. Also, note that Dynamic Time Warping (DTW) is not ideal, as I want similarity at each time point rather than warped distance.

My two questions are:
1) Is it recommended to log-transform the data? I am uncertain at what step in the process I should do this or if it is needed.
2) Currently, I address outliers by restricting points to no more than ± 3 SD above the mean for each time series. Is there an improved way of determining outliers?

In addition, any input on the similarity matching process is appreciated. Does the order of the different similarity steps make sense, or is there anything that should be added/changed?

I appreciate any help. Thank you!

Improve this question
$\endgroup$
1
$\begingroup$

My two questions are: 1) Is it recommended to log-transform the data? I am uncertain at what step in the process I should do this or if it is needed.

Log transforming and taking first differences is an alternative way of data preprocessing. If you want to get rid of the different scales issue and besides make your time series more stationary, you can transform by x[t] / x[t-1]. While log(x[t]) - log(x[t-1]) can be even better in terms of making your resulting timeseries stationary in terms of variance.

2) Currently, I address outliers by restricting points to no more than ± 3 SD above the mean for each time series. Is there an improved way of determining outliers?

If you follow the differentiation trail of preprocessing, you want to keep all time dates in place (if you omit some dates you can screw seasonality and AR parts), so one good way is to:

  • take moving average over neighbor sales data points (centered) or

  • replace your outliers with average values around them (say, taking 1 point at each side).

I hope this gives you helpful hints.

Improve this answer
$\endgroup$
  • $\begingroup$ Alex, thank you! This is very helpful. As a quick clarification, the log-transform and taking first differences can take the place of normalizing (so I don't need to normalize the data), correct? After I do the pre-processing, am I able to apply the log-transform and first-differences to all three of the steps I outlined (Pearson correlation, Engle-Granger, Euclidean distance)? Sorry for the many questions - I really appreciate it! $\endgroup$ – user181973 Oct 27 '17 at 17:20
  • $\begingroup$ You could apply log transform and then differentiate on raw data, which are above or equal to zero only. You cannot, moreover, apply Pearsons on the raw data as you had wanted since the data are non stationary. You can appply correlation to differenced timeseries legally. Other metrics too. $\endgroup$ – Alexey Burnakov Oct 27 '17 at 17:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.