I have a time series representing demand for a product which looks as follows:

enter image description here

Clearly, this time series shows an upward trend and it's variance does seem non-stationary as well.

Further, I have a set of time series for the same timeframe, among which I would like to identify potential early indicators for changes in product demand. The potential indicators are on a very different scale, and also appear "more stationary" both in variance and in trend.

One example of a potential indicator looks as follows:

enter image description here

Now, I would like to start my process of identfying indicators. I have an idea of what I want to do but I am struggling to find the literature to back my ideas. My current plan is the following:

  1. Make product demand series and indicator candidate series stationary (for example through differencing)
  2. Iteratively compare the product demand series with the indicator candidate series' using cross-correlation - looking for lags with high correlation and picking indicators with high cross-correlation at "early" lags

My question is: Is this approach valid? Are there other/better approaches that I could take?

Any help would be greatly appreciated.

Thank you very much in advace!

  • $\begingroup$ Just curious . was my help/response useful. If you are satisfied with my comprehensive answer to your question , please accept it or let me know how else I can help you.. $\endgroup$ – IrishStat Aug 29 '19 at 13:42
  • $\begingroup$ the pw cross corr just suggests a contemporaneous relationship. The pw filter is just the algebraic expansion of the pw arima model. $\endgroup$ – IrishStat Aug 29 '19 at 14:41

you said:

Make product demand series and indicator candidate series stationary (for example through differencing)

I say : Not necessarily as you my be vitiating the importance of the predictor series by pre-empting the effect

Iteratively compare the product demand series with the indicator candidate series' using cross-correlation - looking for leads, lags and contemporaneous effects with high correlation and picking indicators with high cross-correlation using pre-whitening procedures as discussed here Why is prewhitening important? and here The theory behind fitting an ARIMAX model

Post an example of one of your favorite products and I will try and help further .

P.S. in the absence of the causal series your Y series seems to have a level/step change at the last 16 or so observations. Note that a level/step change is in reality an INTERCEPT CHANGE . When you introduce your X predictor it's downwards activity for the latter part of the series might "explain" the level/step shift in Y .


The PRED series that you posted is different from the one you originally included, I detected a zero in it and I remedied it and used it as a candidate to predict Y DEMAND) Here is a plot of PRED enter image description here and a scaled plot of DEMAND vs PRED enter image description here visually suggesting some outside effects , possibly repetitious ...possibly permanent ,,,possibly 1 time only AND a general positive relationship i.e.an upwards trend between SCALED DEMAND and SCALED PRED.

The pre-whitening model for X is here enter image description here suggesting an expanded filter containing 15 coefficients , 10 of which are zero.

The pre-whitened cross-correlation is here suggesting a contemporaneous effect

enter image description here

Intervention Detection suggested adding s few predictors reflecting the "omitted structure" . Here is the equation enter image description here with more detail here enter image description here. The repetitive effects are a negative seasonal pulse for December and a positive level shift starting at July of 2017 ( 17 PERIODS AGO confirming a prior visual finding made before the actual data was received and analyzed) . The non-repetitive anomalies are nearly all visually obvious .

The statistics for this model are here enter image description here

The unusual points were 5 in number reflecting one time anomalies.

Following is the Actual,Fit and Forecast graph enter image description here with residual plot here enter image description here and acf here enter image description here

Finally the Forecast plot is here based upon forecasts generated for the predictor series PRED enter image description here . The confidence intervals include the uncertainty in the predicted PRED and the possibility that future anomalies will be encountered.

In summary there is no need to difference the data in order to build a possibly useful model.

I used an (optionally automated) piece of software [AUTOBOX] that I have helped to develop. This is not meant as an advertisement but as a disclaimer.

Be very aware of the complicated ARIMA portion that was detected reflecting a typical STRONG QUARTERLY EFFECT typical in national account series. Kudos to @chrishmorris for his acumen/eyesight for visually detecting the strong quarterly effect for this monthly series AND SUGGESTING A 3 PERIOD AVERAGE be employed. Note that the value 3 periods ago is optimally weighted by .856 . The final model includes a 3 period weighted average with values .033 for lag 1 , 0. for lag 2 and .856 for lag 3 .

Finally if the OP wishes to specify future values for PRED these are easily used to create alternative forecasts for DEMAND . We show here the 84 historical values for PRED with ARIMA predictions for the next 36 periods. enter image description here

  • $\begingroup$ Thank you very much for your answer. After following the links in in your reply and doing some research, I found this example in which an ARIMA model is fit to x (predictor time series). The coefficient of this ARIMA model is then used to filter y in order to plot a ccf that shows the cross-correlation between the "new" x and the "new" y time series. This seems very much like what I want to do. However I do not understand how one derives the filter c(1,-(1+coef(aa)[1]),coef(aa)[1]) from the model coefficients provided by auto.arima(). $\endgroup$ – servusgude Aug 26 '19 at 14:04
  • $\begingroup$ one need to use software/analysis to express the arima model as a pure right-handside equation. In this case a first difference model with an ar(1) expands to an ar(2) model . In general this expansion often generates a high order ar model. If you post your data I might be able to provide more detail. $\endgroup$ – IrishStat Aug 26 '19 at 15:17
  • $\begingroup$ Thanks, I edited the question to include the data. $\endgroup$ – servusgude Aug 26 '19 at 15:59
  • $\begingroup$ Hi @IrishStat, Thank you very much for your detailed response. Am I right in assuming, that to answer my initial question: "is X a good indicator of Y"? it is enough to follow your analysis until the point where you inspect the "pre-whitened cross-correlation"?. If so, again I am wondering how the pre-whitening filter was derived from the parameters of the pre-whitening model. I am also wondering what type of pre-whitening model you used and what exactly the filter was ( you're mentioning that it should contain "15 coefficients , 10 of which are zero") Thanks again for your help! $\endgroup$ – servusgude Aug 29 '19 at 14:12
  • $\begingroup$ why don't u invite me to chat session so we can continue this , if you can't do this we can communicate via email $\endgroup$ – IrishStat Aug 29 '19 at 14:19

Your demand seems to have a quarterly cycle. I suggest that you separate this out by: - using a three-month sliding window to plot a trend line - calculating average demand on day one, day 2 of the quarter etc to plot the periodic variation - add the two figure and substracting actual demand to determine residuals

This will give you a clearer idea of what is to be explained. Exogenous factors could increase the quarterly variation, change the gradient of the underlying change, or make a temporary change in the size of the residual.

  • $\begingroup$ is this a 3 month centered average with EQUAL weights ? 1/3 , 1/3 and 1/3 or should analysis be used to determine the # of periods to average and the optimal weights that should be applied . $\endgroup$ – IrishStat Aug 23 '19 at 17:10
  • $\begingroup$ J. Shiskin preferred a 13 period moving average with equal weights BBJ ( Before Box & Jenkins ) $\endgroup$ – IrishStat Aug 23 '19 at 17:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.