1
$\begingroup$

For my master thesis I have a dataset with the daily count of orders from a company over ten years. Naturally this data follows strong seasonality with almost no orders on the weekend. To fit an ARIMA model, I hoped to make the data stationary using seasonal differences, but when I plot the differences the ACF and PACF still show significant spikes at the weekly intervals of 7, 14, etc.

library(forecast)
ggtsdisplay(diff(orders, 7), lag.max = 25)

enter image description here

The KPSS and Ljung Box test both tell that the data is stationary, but the plot worries me. I have tried different ways to transform the data:

  • Differencing of higher order
  • BoxCox Transformation

But the plot always shows similar results. Does anyone know the reason for this and how it could be fixed? I have attached a 3-year extract of the data below. As a novice in time series I lack the judgement to tell if it is ok to go ahead and I would highly appreciate your input.

I would need the ARIMA model as a baseline for dynamic regression models as described by Hyndman where I will include day dummies, holidays, months etc. A nice example of what I plan to do is question Arima Model with weekday dummy variables Forecast. Is my assumption correct that with these dummies I no longer need to correct for the seasonality?

Data

data <- c(0L, 1L, 0L, 1L, 50L, 78L, 62L, 65L, 71L, 0L, 0L, 24L, 126L, 
          28L, 60L, 59L, 5L, 0L, 15L, 101L, 43L, 45L, 49L, 0L, 0L, 46L, 
          102L, 107L, 80L, 30L, 1L, 0L, 24L, 30L, 80L, 39L, 64L, 3L, 0L, 
          30L, 37L, 97L, 31L, 17L, 0L, 0L, 12L, 65L, 51L, 21L, 34L, 0L, 
          0L, 35L, 55L, 77L, 50L, 27L, 0L, 0L, 21L, 29L, 63L, 47L, 42L, 
          0L, 0L, 15L, 55L, 65L, 55L, 33L, 0L, 0L, 27L, 37L, 87L, 23L, 
          17L, 1L, 0L, 21L, 22L, 55L, 49L, 41L, 0L, 0L, 11L, 39L, 33L, 
          84L, 1L, 0L, 0L, 2L, 24L, 201L, 45L, 27L, 0L, 0L, 4L, 48L, 52L, 
          65L, 101L, 0L, 0L, 25L, 135L, 65L, 26L, 43L, 0L, 0L, 36L, 66L, 
          23L, 69L, 2L, 0L, 0L, 33L, 45L, 63L, 49L, 31L, 0L, 0L, 46L, 73L, 
          58L, 1L, 11L, 0L, 0L, 14L, 71L, 172L, 60L, 50L, 0L, 0L, 0L, 81L, 
          78L, 58L, 29L, 7L, 0L, 31L, 95L, 83L, 45L, 23L, 0L, 0L, 27L, 
          79L, 59L, 27L, 36L, 0L, 0L, 60L, 45L, 47L, 18L, 5L, 0L, 0L, 53L, 
          60L, 45L, 14L, 33L, 0L, 0L, 35L, 75L, 40L, 42L, 78L, 0L, 0L, 
          8L, 71L, 42L, 65L, 30L, 0L, 0L, 32L, 73L, 78L, 69L, 36L, 0L, 
          0L, 55L, 46L, 27L, 78L, 79L, 0L, 0L, 26L, 46L, 39L, 58L, 13L, 
          0L, 0L, 40L, 43L, 32L, 12L, 18L, 0L, 0L, 6L, 21L, 89L, 74L, 19L, 
          0L, 0L, 17L, 44L, 55L, 44L, 18L, 0L, 0L, 33L, 45L, 59L, 39L, 
          24L, 0L, 0L, 46L, 19L, 59L, 27L, 61L, 1L, 0L, 15L, 55L, 37L, 
          24L, 36L, 0L, 0L, 5L, 25L, 43L, 34L, 39L, 0L, 0L, 30L, 82L, 22L, 
          34L, 24L, 0L, 0L, 27L, 39L, 42L, 23L, 32L, 0L, 0L, 40L, 73L, 
          20L, 26L, 14L, 0L, 0L, 44L, 38L, 43L, 29L, 42L, 0L, 0L, 31L, 
          81L, 41L, 28L, 20L, 0L, 0L, 51L, 62L, 19L, 30L, 21L, 0L, 0L, 
          23L, 88L, 101L, 72L, 35L, 0L, 0L, 14L, 58L, 65L, 56L, 40L, 0L, 
          0L, 25L, 118L, 13L, 56L, 29L, 0L, 0L, 11L, 51L, 43L, 41L, 40L, 
          0L, 0L, 78L, 65L, 25L, 23L, 81L, 0L, 0L, 30L, 22L, 46L, 78L, 
          29L, 0L, 0L, 58L, 38L, 41L, 38L, 57L, 0L, 0L, 27L, 40L, 91L, 
          9L, 0L, 0L, 0L, 20L, 47L, 3L, 7L, 0L, 0L, 0L, 17L, 50L, 41L, 
          15L, 95L, 0L, 0L, 24L, 59L, 77L, 54L, 32L, 6L, 0L, 35L, 81L, 
          38L, 53L, 21L, 1L, 0L, 69L, 52L, 104L, 34L, 24L, 12L, 0L, 37L, 
          20L, 120L, 58L, 17L, 0L, 0L, 42L, 82L, 79L, 38L, 24L, 0L, 0L, 
          37L, 67L, 64L, 50L, 23L, 0L, 0L, 17L, 21L, 25L, 52L, 46L, 0L, 
          0L, 40L, 58L, 111L, 20L, 19L, 3L, 0L, 58L, 45L, 74L, 38L, 32L, 
          0L, 0L, 17L, 15L, 64L, 29L, 41L, 1L, 0L, 37L, 66L, 38L, 23L, 
          0L, 0L, 0L, 4L, 56L, 97L, 36L, 31L, 0L, 0L, 23L, 41L, 54L, 19L, 
          43L, 0L, 0L, 63L, 32L, 63L, 85L, 79L, 0L, 0L, 11L, 60L, 90L, 
          39L, 29L, 0L, 0L, 78L, 13L, 94L, 51L, 66L, 0L, 0L, 20L, 19L, 
          44L, 1L, 11L, 0L, 0L, 27L, 33L, 78L, 48L, 38L, 0L, 0L, 1L, 15L, 
          37L, 64L, 43L, 0L, 0L, 25L, 67L, 106L, 7L, 29L, 0L, 0L, 87L, 
          64L, 56L, 37L, 42L, 1L, 0L, 19L, 62L, 45L, 65L, 41L, 0L, 0L, 
          19L, 33L, 66L, 17L, 35L, 0L, 0L, 82L, 35L, 75L, 55L, 23L, 0L, 
          0L, 21L, 54L, 23L, 17L, 17L, 0L, 0L, 15L, 68L, 75L, 31L, 80L, 
          0L, 0L, 5L, 114L, 69L, 21L, 46L, 0L, 0L, 49L, 34L, 66L, 12L, 
          32L, 0L, 0L, 20L, 66L, 33L, 67L, 37L, 0L, 0L, 4L, 49L, 42L, 59L, 
          31L, 0L, 0L, 33L, 40L, 54L, 79L, 56L, 0L, 0L, 29L, 18L, 44L, 
          44L, 27L, 0L, 0L, 28L, 15L, 147L, 52L, 24L, 0L, 0L, 14L, 85L, 
          47L, 38L, 53L, 4L, 0L, 37L, 63L, 30L, 29L, 52L, 0L, 0L, 6L, 80L, 
          16L, 50L, 42L, 0L, 0L, 28L, 27L, 118L, 29L, 52L, 2L, 0L, 13L, 
          38L, 24L, 31L, 83L, 0L, 0L, 12L, 25L, 64L, 32L, 26L, 0L, 0L, 
          45L, 86L, 42L, 73L, 29L, 0L, 0L, 14L, 52L, 85L, 27L, 60L, 0L, 
          0L, 18L, 33L, 35L, 62L, 16L, 0L, 0L, 13L, 17L, 42L, 57L, 41L, 
          1L, 0L, 53L, 53L, 129L, 95L, 14L, 3L, 0L, 21L, 59L, 30L, 57L, 
          63L, 0L, 0L, 12L, 70L, 46L, 13L, 64L, 0L, 0L, 10L, 25L, 98L, 
          28L, 20L, 0L, 0L, 72L, 54L, 32L, 27L, 66L, 0L, 0L, 20L, 40L, 
          72L, 42L, 57L, 0L, 0L, 37L, 35L, 66L, 47L, 99L, 2L, 0L, 0L, 57L, 
          59L, 20L, 6L, 0L, 0L, 4L, 21L, 9L, 39L, 44L, 0L, 0L, 17L, 19L, 
          21L, 47L, 38L, 1L, 0L, 19L, 56L, 112L, 44L, 34L, 0L, 0L, 85L, 
          107L, 47L, 13L, 47L, 0L, 0L, 100L, 94L, 44L, 82L, 27L, 0L, 0L, 
          53L, 25L, 158L, 35L, 46L, 0L, 0L, 52L, 112L, 104L, 69L, 116L, 
          0L, 0L, 82L, 56L, 74L, 49L, 52L, 0L, 1L, 22L, 25L, 60L, 64L, 
          44L, 7L, 2L, 13L, 93L, 131L, 37L, 78L, 0L, 0L, 8L, 46L, 71L, 
          32L, 20L, 0L, 0L, 17L, 76L, 42L, 33L, 33L, 0L, 0L, 49L, 29L, 
          75L, 25L, 12L, 10L, 0L, 22L, 81L, 74L, 39L, 25L, 0L, 0L, 16L, 
          60L, 30L, 48L, 1L, 0L, 0L, 1L, 89L, 32L, 53L, 35L, 1L, 0L, 10L, 
          42L, 33L, 30L, 59L, 0L, 0L, 0L, 58L, 34L, 57L, 51L, 2L, 0L, 75L, 
          54L, 75L, 33L, 26L, 0L, 0L, 56L, 72L, 104L, 90L, 31L, 3L, 0L, 
          9L, 70L, 40L, 2L, 30L, 0L, 0L, 64L, 85L, 61L, 54L, 38L, 33L, 
          0L, 3L, 45L, 13L, 32L, 45L, 0L, 0L, 57L, 34L, 44L, 55L, 44L, 
          0L, 1L, 16L, 124L, 53L, 71L, 28L, 0L, 0L, 40L, 38L, 39L, 52L, 
          74L, 18L, 0L, 27L, 59L, 35L, 81L, 58L, 0L, 0L, 33L, 66L, 95L, 
          51L, 27L, 4L, 0L, 15L, 33L, 53L, 59L, 24L, 0L, 0L, 25L, 61L, 
          114L, 57L, 22L, 0L, 0L, 22L, 16L, 41L, 28L, 24L, 0L, 0L, 29L, 
          39L, 61L, 38L, 13L, 0L, 0L, 34L, 64L, 69L, 53L, 18L, 0L, 0L, 
          28L, 68L, 68L, 47L, 23L, 0L, 0L, 24L, 16L, 60L, 33L, 27L, 0L, 
          0L, 28L, 59L, 70L, 30L, 41L, 11L, 0L, 7L, 22L, 126L, 39L, 64L, 
          2L, 0L, 20L, 45L, 26L, 55L, 63L, 0L, 0L, 10L, 41L, 22L, 51L, 
          7L, 0L, 0L, 13L, 42L, 59L, 49L, 25L, 2L, 1L, 11L, 79L, 29L, 40L, 
          16L, 4L, 0L, 29L, 24L, 47L, 77L, 91L, 0L, 37L, 15L, 28L, 24L, 
          75L, 22L, 1L, 21L, 41L, 20L, 40L, 58L, 53L, 4L, 0L, 18L, 45L, 
          64L, 40L, 33L, 4L, 0L, 39L, 68L, 79L, 22L, 33L, 46L, 0L, 13L, 
          26L, 50L, 90L, 25L, 0L, 0L, 26L, 16L, 94L, 48L, 51L, 1L, 0L, 
          147L, 95L, 30L, 35L, 69L, 0L, 0L, 27L, 67L, 66L, 37L, 45L, 1L, 
          0L, 41L, 36L, 70L, 11L, 29L, 31L, 0L, 0L, 1L, 24L, 63L, 38L, 
          0L, 0L, 0L, 12L, 31L, 33L, 30L, 0L, 0L, 33L, 27L, 19L, 31L, 47L, 
          27L, 0L, 21L, 86L, 74L, 55L, 36L, 58L, 0L, 37L, 36L, 49L, 35L, 
          58L, 0L, 0L, 49L, 55L, 60L, 60L, 67L, 32L, 0L, 25L, 35L, 75L, 
          79L, 65L, 3L, 0L, 60L, 51L, 112L, 33L, 80L, 3L, 0L, 15L, 94L, 
          20L, 33L, 16L, 40L, 0L, 18L, 41L, 76L, 42L, 39L, 0L, 58L, 72L, 
          11L, 23L, 48L, 76L, 24L, 0L, 17L, 57L, 48L, 42L, 96L, 18L, 0L, 
          11L, 48L, 22L, 77L, 15L, 0L, 0L, 30L, 37L, 79L, 43L, 1L, 0L, 
          0L, 0L, 64L, 23L, 29L, 42L, 0L, 0L, 15L, 31L, 48L, 135L, 36L, 
          39L, 0L, 9L, 39L, 38L, 59L, 44L, 14L, 0L, 57L, 99L, 19L, 59L, 
          28L, 0L, 0L, 43L, 0L, 27L, 43L, 72L, 0L, 0L, 71L, 40L, 33L, 0L, 
          13L, 2L, 0L, 42L, 93L, 90L, 40L, 32L, 0L, 6L, 0L, 28L, 32L, 33L, 
          46L, 0L, 0L, 24L, 73L, 104L, 43L, 74L, 0L, 0L, 41L, 48L, 81L, 
          66L, 36L, 0L, 0L, 18L, 37L, 41L, 26L, 22L, 29L, 0L, 27L, 35L, 
          21L, 22L, 37L, 0L, 0L, 23L, 49L, 54L, 68L, 34L, 1L, 0L, 37L, 
          55L, 21L, 32L, 88L, 0L, 0L, 41L, 60L, 73L, 52L, 16L, 1L, 0L, 
          16L, 34L, 66L, 68L, 32L, 3L, 0L, 27L, 32L, 60L, 73L, 26L, 0L, 
          0L, 17L, 28L, 12L, 75L, 49L, 0L, 0L, 100L, 53L, 57L, 60L, 18L, 
          0L, 0L, 17L, 51L, 75L, 29L, 23L, 0L, 0L, 60L, 19L, 20L, 43L, 
          39L, 0L, 0L, 22L, 52L, 69L, 24L, 12L, 0L, 0L, 14L, 21L, 42L, 
          107L, 47L, 11L, 0L, 6L, 52L, 45L, 78L, 98L, 0L, 1L, 20L, 44L, 
          31L, 52L, 29L, 0L, 0L, 13L, 57L, 52L, 47L, 91L, 0L, 0L, 31L, 
          31L, 45L, 10L, 30L, 0L, 0L, 37L, 33L, 54L, 42L, 49L, 0L, 2L, 
          29L, 63L, 42L, 47L, 43L, 2L, 0L, 14L, 36L, 48L, 102L, 66L, 0L, 
          0L, 25L, 87L, 154L, 24L, 19L, 0L, 0L, 31L, 47L, 50L, 42L, 25L, 
          116L, 0L, 47L, 29L, 26L, 76L, 32L, 19L, 72L, 17L, 23L, 36L, 32L, 
          51L, 17L, 0L, 42L, 32L, 23L, 66L, 18L)

orders <- ts(data, frequency = 365.25, start = c(2015,1))
Improve this question
$\endgroup$
6
  • $\begingroup$ take a look at some of these recent posts to get a perspective on how to deal with memory and latent deterministic structure in daily data stats.stackexchange.com/… $\endgroup$
    – IrishStat
    Commented May 11, 2019 at 10:44
  • $\begingroup$ stats.stackexchange.com/questions/399570/… suggests that arima be added after important/significant latent deterministic structure has been included in the model $\endgroup$
    – IrishStat
    Commented May 11, 2019 at 12:01
  • $\begingroup$ @IrishStat thank you very much for the fast response. I have reviewed these posts and found that I actually read some of them. The features you listed in the second comment are highly relevant, I will definitely use them in a dynamic regression. Nevertheless: can I use the one-season-differenced data from above for a baseline-ARIMA model or do you think with this data a dynamic regression is the only way? $\endgroup$
    – Patrick Glettig
    Commented May 11, 2019 at 13:11
  • $\begingroup$ Definitely not .... It is much safer to identify and introduce latent determinstic structure while adjusting for pulses ( a latent deterministic structure ) before introducing differencing or an arima structure. Differencing is one klind of adjustment for a non-stationary series ...yet another is to de-mean (introduce needed level shift indicators) see stats.stackexchange.com/search?q=user%3A3382+de-mean $\endgroup$
    – IrishStat
    Commented May 11, 2019 at 13:51
  • $\begingroup$ if you email me the data in a csv format I will try and see if I can help you further . include the starting date and the name of the country . Make sure you have an observation for every day ...even if it is a zero $\endgroup$
    – IrishStat
    Commented May 11, 2019 at 14:27

1 Answer 1

Reset to default
1
$\begingroup$

I took your 3925 days ( 3/3/2008 - 11/30/2018 )of orders and introduced the data enter image description here to AUTOBOX. It found a useful model that included 2 level shifts , 4 holidays , Monday-after-a-holiday effect, 3 months of the year (note that month 1 is March ..months found significant were July, Oct,and Feb ALL low ) , 5 significant daily factors AND two days of the month where orders are statistically different ( day 1, day 2 ) . There is no need for any ARIMA as the residuals from this "causative model " show no evidence of structure/predictability.

Here is the Actual , Fit and Forecast enter image description here . The Forecasts for the next 365 days are here enter image description here

The equation is shown here in sections enter image description here and enter image description here

The model is also shown here enter image description here and enter image description here and enter image description here and terminally here enter image description here

The model residuals plot is here enter image description here with acf here ( note the large sample size imposes unwarranted/ridiculous tight limits enter image description here

In summary the level shifts are subtle but significant . Thus to answer your question as to how to make this series stationary , simply take into account the level shifts ( by de-meaning) and also take into account the temporal variables that have been enumerated here. By the way your suggestion of just using 1 variable to deal with holiday effects only works if and only if the response is uniform across all all holidays.

Hope this helps you and others who are faced with the daunting task of extracting signal from data sets like this. Remember all models are rong BUT some are useful (GEPB) .

Finally the model statistics are here enter image description here

Improve this answer
$\endgroup$
3
  • $\begingroup$ Wow, thank you so much! There is plenty of interesting insights from this. I would like to ask two things for clarification: by demeaning the data, do you simply mean subtracting the average for periods in the 3 different levels? What technique can I use to discover the two model shifts? $\endgroup$
    – Patrick Glettig
    Commented May 12, 2019 at 6:10
  • $\begingroup$ essentially yes or simply include two indicator series (level shifts) as predictors. Detecting level shifts can sometimes (not in this case however) be done with the R program tsoutliers cran.r-project.org/web/packages/tsoutliers/tsoutliers.pdf or AUTOBOX ( which is available in R) and which I have helped to write. In general the technnique is called Intervention Detection docplayer.net/… $\endgroup$
    – IrishStat
    Commented May 12, 2019 at 6:53
  • $\begingroup$ In summary, you would now NOT NEED an ARMA model or any differencing as the residuals FROM A causative model ARE approximately WHITE NOISE. The suggested causative model includes mean shifts and other temporal variables such as before and after 4 holidays ,day-of-the-week, month-of-the-year,. day-of-the-month , monday-after-a-holiday $\endgroup$
    – IrishStat
    Commented May 12, 2019 at 9:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.