How to model timeseries with unequally-spaced seasonality interval

Question

I have a timeseries that has an irregular seasonality interval, and two seasonalities.

The data is basically daily data, with one weekly regular seasonality, and the other is a spike at the beginning of each month. Since the data is daily, and months have unequal number of days, the seasonality component cannot be specified.

I am interested in generating both the prediction and confidence intervals over the new data.

Is there a proper way to handle this situation? Please point me to methods/papers that handle this situation.

The way I have come up with is to fit the weekly seasonality. And then from the residuals, take out values of first of each month, and fit a separate model on it. And then the prediction will be sum of outputs of two models, and confidence intervals can be summed. Is this approach correct/reasonable?

Update

Based on one of the answers, I would like to add a sample data in the form of R code. Please run this code, it generates data similar to what I am seeing. There are two seasonalities, weekly is fixed, and monthly is irregular due to unequal number of days in a month:

startDate = as.Date("2015-01-01")
numDays = 365

dateRange = seq(0,numDays - 1)
dates = startDate + dateRange

fixedTrend = 0.075
fixedTrendSD = 0.05

startOfMonthExtra = 40
startOfMonthExtraSD = 10

set.seed(400)
initialWeekData = rnorm(7, 20, 2.5)
dailySD = 1.5

finalData = data.frame(index = (dateRange + 1), date = dates, transactions = rep(0, length(dateRange)))

for (index in dateRange)
{
  numTrans = (initialWeekData[(index %% 7) + 1] + rnorm(1, 0, dailySD)) +
    ((fixedTrend * index) + rnorm(1, mean = 0, sd = fixedTrendSD))


  if (as.numeric(format(finalData$date[index + 1], '%d')) == 1)
  {
    numTrans = numTrans + (startOfMonthExtra + rnorm(1, mean = 0, sd = startOfMonthExtraSD))
  }

  finalData$transactions[index + 1] = numTrans
}

plot(finalData$transactions, type = 'l')
plot(acf(finalData$transactions, lag.max = 100))
plot(pacf(finalData$transactions))
print (finalData)

General Trend:

Autocorrelation:

Data

Here is the data, copy-pasted from .csv export:

date,transactions
1/1/2015,52.74906468
1/2/2015,20.26490186
1/3/2015,21.43729499
1/4/2015,16.7894963
1/5/2015,17.79133722
1/6/2015,17.78910233
1/7/2015,23.95933873
1/8/2015,17.90526239
1/9/2015,24.5390435
1/10/2015,26.05331654
1/11/2015,19.67518334
1/12/2015,18.99410084
1/13/2015,18.23845702
1/14/2015,22.27144548
1/15/2015,16.72718262
1/16/2015,22.72594348
1/17/2015,27.32192626
1/18/2015,21.17935174
1/19/2015,18.66000735
1/20/2015,18.87345238
1/21/2015,21.17792699
1/22/2015,18.15240481
1/23/2015,23.52586716
1/24/2015,22.01550349
1/25/2015,19.79979164
1/26/2015,19.19412584
1/27/2015,17.06655018
1/28/2015,25.50157375
1/29/2015,19.32727694
1/30/2015,24.14161956
1/31/2015,26.11765094
2/1/2015,59.93922667
2/2/2015,20.7584508
2/3/2015,17.63204915
2/4/2015,26.67839512
2/5/2015,19.57360293
2/6/2015,25.00709686
2/7/2015,26.4067203
2/8/2015,19.74685651
2/9/2015,25.10919485
2/10/2015,18.49668817
2/11/2015,24.41458171
2/12/2015,24.635849
2/13/2015,25.51776589
2/14/2015,26.7552656
2/15/2015,21.56235616
2/16/2015,24.57824132
2/17/2015,20.79233479
2/18/2015,27.74442842
2/19/2015,24.35732249
2/20/2015,25.32933146
2/21/2015,27.40369665
2/22/2015,21.03439117
2/23/2015,21.46189783
2/24/2015,25.15134398
2/25/2015,26.58888224
2/26/2015,21.88857894
2/27/2015,23.28575392
2/28/2015,28.17492282
3/1/2015,61.04047234
3/2/2015,22.52411894
3/3/2015,23.39449923
3/4/2015,25.3298269
3/5/2015,20.93176067
3/6/2015,24.87645721
3/7/2015,30.02425267
3/8/2015,22.32884652
3/9/2015,25.29222318
3/10/2015,22.70396971
3/11/2015,28.94134728
3/12/2015,22.48180209
3/13/2015,28.16091043
3/14/2015,30.08662193
3/15/2015,24.25983217
3/16/2015,24.1430691
3/17/2015,19.28079169
3/18/2015,24.8525012
3/19/2015,21.10902186
3/20/2015,27.43610192
3/21/2015,27.87100596
3/22/2015,25.10246144
3/23/2015,25.60089595
3/24/2015,23.14736502
3/25/2015,27.72033492
3/26/2015,23.86414591
3/27/2015,27.77950546
3/28/2015,30.98609311
3/29/2015,23.53327717
3/30/2015,26.68881454
3/31/2015,23.87447232
4/1/2015,62.59991258
4/2/2015,22.21949117
4/3/2015,31.16045633
4/4/2015,33.32556059
4/5/2015,23.12779659
4/6/2015,24.17145667
4/7/2015,23.22960796
4/8/2015,27.76120857
4/9/2015,21.36272363
4/10/2015,29.73239776
4/11/2015,33.89432951
4/12/2015,23.95673011
4/13/2015,26.48769295
4/14/2015,23.51676258
4/15/2015,29.87881956
4/16/2015,25.39996185
4/17/2015,27.77134841
4/18/2015,32.87153279
4/19/2015,23.82516631
4/20/2015,23.40578959
4/21/2015,26.32813244
4/22/2015,33.5125293
4/23/2015,25.20142231
4/24/2015,28.91069682
4/25/2015,31.36142016
4/26/2015,25.57649463
4/27/2015,27.48794724
4/28/2015,30.21320199
4/29/2015,31.04575019
4/30/2015,28.02868167
5/1/2015,88.30220545
5/2/2015,31.22167308
5/3/2015,27.79118008
5/4/2015,28.92808079
5/5/2015,25.62267133
5/6/2015,34.35547234
5/7/2015,25.28690905
5/8/2015,32.28412738
5/9/2015,33.00552145
5/10/2015,28.71009173
5/11/2015,27.89390656
5/12/2015,23.00851188
5/13/2015,32.51427658
5/14/2015,28.31453993
5/15/2015,32.48612835
5/16/2015,36.72525409
5/17/2015,30.55136993
5/18/2015,30.59941076
5/19/2015,26.63919888
5/20/2015,32.98538709
5/21/2015,29.90209919
5/22/2015,30.04983953
5/23/2015,33.9988683
5/24/2015,25.25469139
5/25/2015,30.39534307
5/26/2015,26.56368494
5/27/2015,30.51238564
5/28/2015,28.9254327
5/29/2015,30.71114583
5/30/2015,34.78696811
5/31/2015,24.94027415
6/1/2015,74.28754259
6/2/2015,26.59407057
6/3/2015,33.22002922
6/4/2015,28.84305444
6/5/2015,33.3375238
6/6/2015,37.09151009
6/7/2015,28.85770066
6/8/2015,31.00777145
6/9/2015,26.76342588
6/10/2015,33.57435781
6/11/2015,28.94518456
6/12/2015,33.89546877
6/13/2015,34.57983147
6/14/2015,31.32341617
6/15/2015,31.48026864
6/16/2015,26.28231039
6/17/2015,35.89480482
6/18/2015,31.7865621
6/19/2015,35.37461368
6/20/2015,38.79539265
6/21/2015,34.25980202
6/22/2015,31.57309058
6/23/2015,30.37720087
6/24/2015,35.93092659
6/25/2015,29.78481191
6/26/2015,35.06005626
6/27/2015,37.75122007
6/28/2015,31.76659805
6/29/2015,32.24146208
6/30/2015,30.54196547
7/1/2015,75.24837482
7/2/2015,32.14945138
7/3/2015,35.74500193
7/4/2015,38.52532998
7/5/2015,31.39381869
7/6/2015,29.51083179
7/7/2015,31.36472229
7/8/2015,35.51598092
7/9/2015,33.17465526
7/10/2015,34.70829876
7/11/2015,37.6245739
7/12/2015,32.14475608
7/13/2015,35.0500712
7/14/2015,30.19776271
7/15/2015,33.94666676
7/16/2015,33.92506816
7/17/2015,34.41342329
7/18/2015,36.93721773
7/19/2015,32.74381964
7/20/2015,36.41749619
7/21/2015,32.72610068
7/22/2015,39.44918809
7/23/2015,32.52347484
7/24/2015,37.87427753
7/25/2015,40.25339044
7/26/2015,30.7177176
7/27/2015,34.78966262
7/28/2015,31.12420006
7/29/2015,36.46046884
7/30/2015,29.28277298
7/31/2015,35.99929407
8/1/2015,74.48560726
8/2/2015,35.63800631
8/3/2015,35.47490199
8/4/2015,29.36688537
8/5/2015,37.21354314
8/6/2015,33.80245219
8/7/2015,37.94717398
8/8/2015,43.05034599
8/9/2015,35.77011648
8/10/2015,36.0386634
8/11/2015,32.42831957
8/12/2015,40.07310824
8/13/2015,34.73862784
8/14/2015,38.03008273
8/15/2015,42.04245069
8/16/2015,36.48571884
8/17/2015,37.50188326
8/18/2015,32.45288796
8/19/2015,38.95330014
8/20/2015,36.86774375
8/21/2015,39.82742352
8/22/2015,41.68181886
8/23/2015,35.22646214
8/24/2015,35.00545894
8/25/2015,34.72780964
8/26/2015,38.23701977
8/27/2015,34.63210292
8/28/2015,39.66083695
8/29/2015,40.42898606
8/30/2015,32.1471454
8/31/2015,38.21191106
9/1/2015,93.50325823
9/2/2015,39.46874782
9/3/2015,36.8771921
9/4/2015,41.98774396
9/5/2015,42.08417058
9/6/2015,35.53847934
9/7/2015,37.64036549
9/8/2015,33.20758963
9/9/2015,40.92092121
9/10/2015,37.42273195
9/11/2015,41.13518481
9/12/2015,43.82840339
9/13/2015,35.78505457
9/14/2015,35.04542522
9/15/2015,34.8761835
9/16/2015,41.81141484
9/17/2015,36.44777297
9/18/2015,42.13082146
9/19/2015,44.03418836
9/20/2015,37.63946626
9/21/2015,39.03561644
9/22/2015,35.66483975
9/23/2015,39.20212788
9/24/2015,39.70226338
9/25/2015,40.35919028
9/26/2015,42.20599185
9/27/2015,39.77900844
9/28/2015,38.79582864
9/29/2015,34.86517012
9/30/2015,41.83860638
10/1/2015,64.94589366
10/2/2015,43.07562142
10/3/2015,47.64426728
10/4/2015,38.66153481
10/5/2015,38.37707182
10/6/2015,35.41071995
10/7/2015,42.81257638
10/8/2015,36.75442644
10/9/2015,41.11838625
10/10/2015,43.26054093
10/11/2015,39.33529341
10/12/2015,41.6202536
10/13/2015,39.2937663
10/14/2015,45.13837071
10/15/2015,37.68135686
10/16/2015,42.28885751
10/17/2015,46.09535885
10/18/2015,41.09881128
10/19/2015,38.96142984
10/20/2015,37.4262952
10/21/2015,44.11265484
10/22/2015,38.10922703
10/23/2015,45.17073062
10/24/2015,43.01451162
10/25/2015,38.4376718
10/26/2015,39.82727262
10/27/2015,38.53620385
10/28/2015,42.71124829
10/29/2015,40.15332283
10/30/2015,45.42560913
10/31/2015,45.32991473
11/1/2015,85.19126887
11/2/2015,41.17704501
11/3/2015,41.87762019
11/4/2015,46.56773754
11/5/2015,40.39743425
11/6/2015,46.30024185
11/7/2015,51.42270834
11/8/2015,41.84595701
11/9/2015,41.19274366
11/10/2015,41.3061038
11/11/2015,47.59450651
11/12/2015,41.51138025
11/13/2015,44.26365645
11/14/2015,45.71445966
11/15/2015,40.38499539
11/16/2015,41.5100277
11/17/2015,40.94941586
11/18/2015,47.7093646
11/19/2015,43.84913342
11/20/2015,48.25256602
11/21/2015,50.29579342
11/22/2015,43.22157278
11/23/2015,44.86271062
11/24/2015,40.26497057
11/25/2015,46.65985501
11/26/2015,42.93185837
11/27/2015,45.87778224
11/28/2015,50.60941551
11/29/2015,44.47744372
11/30/2015,43.56422373
12/1/2015,72.68901096
12/2/2015,46.01277718
12/3/2015,43.64688957
12/4/2015,44.65016888
12/5/2015,49.89217889
12/6/2015,43.773928
12/7/2015,44.02364
12/8/2015,44.67308344
12/9/2015,47.06286599
12/10/2015,42.04756714
12/11/2015,47.87248763
12/12/2015,50.37456947
12/13/2015,44.16158594
12/14/2015,40.66425086
12/15/2015,43.71989532
12/16/2015,48.31577609
12/17/2015,43.03672144
12/18/2015,46.35942783
12/19/2015,49.90144912
12/20/2015,44.61480031
12/21/2015,45.34092281
12/22/2015,45.87761193
12/23/2015,49.99266589
12/24/2015,46.36219303
12/25/2015,49.82884424
12/26/2015,51.04211724
12/27/2015,45.23018832
12/28/2015,44.69989931
12/29/2015,39.53219808
12/30/2015,50.89079503
12/31/2015,45.7846928

Answer 1

What you need to do is to incorporate day-of-the-week , changes in day-of-the-week effects , level shift effects , local time trends , specific days-of-the-month efffects, weekly effects, monthly effects, week-of-the-month effects , pre, contemporay and lag effects of holidays/events, ARIMA memory , long-weekend effects as needed while dealing with anomalous one-time effects and possible error variance effects . Take a look at http://www.autobox.com/cms/index.php/afs-university/intro-to-forecasting/doc_download/53-capabilities-presentation for some material on daily modelling. For example just yesterday I was reviewing some output for a daily series analysis that you may find interesting. If you wish to post your data in a single columnn csv file with starting date and country , I will try and take a look at it to help you sort out what can be done. In my opinion your well-intended but ad hoc piecemeal approach is just not going to provide you with a meaningful analysis. You can send your data directly to me in case you can't post it to the group. It would be better to post your data and give other readers an opportunity to feel your pain as you attempt to extract information by identifying a model.

EDITED AFTER ANALYSIS OF DATA

Since only 365 days was available no monthly effects. weekly effects , holiday effects , long-weekend effects et. al. could be identified ..

I used one of my favorite toys (AUTOBOX which I helped to develop) and obtained what appears to be a very reasonable model. I plotted the data (365 daily values for 1 year) and AUTOBOX presented the ACF of the original series visually suggesting non-stationarity. AUTOBOX conducted a fairly exhaustive search process and concluded that the dominant model for this data include two trends and a day 1 effect . Note that this was done in a totally automatic way and uses heuristics crafted over some 48 years of development. It further found that day1 was both consistently unusual but also periodically inconsistent and constructed pulse (0/1) series to remedy the resultant inconsistancy. The final model yielded an Actual/Fit and Forecast graph and a graph of the forecasts here

The FIXED_EFF variables reflect day-of-the week indicators where apparently DAY2 ( by it's omission )is not different from the overall average where DAY01,DAY03,DAY04,DAY05 and DAY06 are.

The FIXED_DAY01 variable is a 0/1 variable with a "1" on day one of the month and a "0" elsewhere. This variable was not suggested by me but rather "discovered" by AUTOBOX.

The two trend series I~T reflect the breeak in the overall trend at time period 42.

The I~P variables reflect simple one-time pulses for a particualr day in the year.

The model is presented here and partially here

Thorough statsistical requires some validation that the model is sufficient thus we present a plot of the final residuals and the ACF of the model errors . All parameters are needed (statistically significant) and test for automatically selecting a power transform yielded negative results.

Now as you requested .... you want a script to do these things

1. write a script of your choosing to deteremine if there are trends or level shifts in your data
2. write a script to determine what days of the month have a significant effect
3. write a script to detect any anomalies/outliers for all time points
4. write a script to generate a family of forecasts via monte carlo which will generate forecast distributions/confidence intervals to reflect the possibility of future anomalies arising

Following is an example of the script that you can use to run AUTOBOX under R without you having to write the previous 4 scripts.

5)

Move to the autoboxR directory.

#setwd("C:/Users/jin/Desktop/Autobox/AutoboxR") setwd("~/autoboxR")

Load the autobox library.

library(autobox)

Initialize the autobox package.

autoboxInitPackage()

#iErrorCode <- autoboxCheckLicense()

file=read.csv("saltlakedailyshipment.csv") file=file[,3] View(file) dim(file) length(file)

Create a matrix with one column of file value

#########################################frequency=7 demand <- matrix(file, ncol=1, dimnames=list(c(1:length(file)),c("__010115demand"))) demand.ts=ts(demand,start=c(2008,1,1),frequency=7)

Make the file as a autobox object.

demand.autobox <- autobox(demand.ts, iDataType=c(0), iObjective=c(0,0,0), iNumberOfRetainedValues=0, iNumberOfForecastValues=100, cPath=".\Output") demand.autobox.cal=autoboxRun(demand.autobox) ###print the result autoboxPrint(demand.autobox.cal,selection="acefhirt") ###plot the result autoboxPlot(demand.autobox.cal,selection="fh")

For the first forecast period

For the second Forecast Period

Answer 2

I'd try 6 dummy variables for the days of the week, e.g. Sunday as a base and dummies for Monday, Tuesday ... Saturday.

And then a dummy for The beginning of the month, the structure of this depending on whether this is just the first day of the month or whether it's the first week, etc.