ARIMA model has trouble forecasting next month

Question

I am implementing a statsmodels.tsa.arima_model.ARIMA model with daily advertising spend data. I have a dataset that ranges from Jan 1 2016 - present. Using a rolling forecast, I am able to generate fairly accurate next-day predictions.

However, when my training data reaches EOM, the model has trouble predicting the next month's daily spend. Empirically, there is a spike in daily spend at the end of each month, with a significant drop off when the new month begins.

I'm using a 19 day lag period with first order differences.

I figured the model would account for these periodic dips. I've considered adding an observation for the first day of the next month, but have no idea how to estimate this or if this is even a good idea. I'd prefer not to extrapolate data points in the future.

Any help would be greatly appreciated.

Data set:

"Date","Spend"
"01-01-2016",56535.01376
"01-02-2016",55660.69725
"01-03-2016",56686.15915
"01-04-2016",80700.27466
"01-05-2016",80920.13555
"01-06-2016",87146.8539 
"01-07-2016",93110.4974 
"01-08-2016",92782.48688
"01-09-2016",89894.08987
"01-10-2016",89767.35218
"01-11-2016",97125.22324
"01-12-2016",97789.22612
"01-13-2016",85715.50747
"01-14-2016",108069.4139
"01-15-2016",106000.7602
"01-16-2016",102035.6198
"01-17-2016",103513.2055
"01-18-2016",109261.448 
"01-19-2016",111945.1518
"01-20-2016",119439.1549
"01-21-2016",128328.0499
"01-22-2016",140395.7647
"01-23-2016",136612.1674
"01-24-2016",134119.4995
"01-25-2016",154105.4395
"01-26-2016",161823.3877
"01-27-2016",158737.2607
"01-28-2016",163533.6454
"01-29-2016",159264.5691
"01-30-2016",135183.7079
"01-31-2016",116309.2296
"02-01-2016",119186.1448
"02-02-2016",117226.9433
"02-03-2016",119298.8206
"02-04-2016",121761.0795
"02-05-2016",126521.7961
"02-06-2016",121919.0252
"02-07-2016",119699.4814
"02-08-2016",121846.9216
"02-09-2016",131565.4512
"02-10-2016",142011.6188
"02-11-2016",145375.0653
"02-12-2016",168050.0243
"02-13-2016",161023.5091
"02-14-2016",165631.6513
"02-15-2016",146212.1473
"02-16-2016",151513.9778
"02-17-2016",157069.1047
"02-18-2016",170031.9771
"02-19-2016",185936.0284
"02-20-2016",183981.8713
"02-21-2016",178688.9548
"02-22-2016",191702.9579
"02-23-2016",176389.2076
"02-24-2016",176037.8512
"02-25-2016",189056.0767
"02-26-2016",192843.7637
"02-27-2016",165919.0125
"02-28-2016",155907.6034
"02-29-2016",138976.8326
"03-01-2016",116542.9956
"03-02-2016",132595.9346
"03-03-2016",127940.946 
"03-04-2016",130171.0442
"03-05-2016",130965.478 
"03-06-2016",138010.3392
"03-07-2016",139858.3831
"03-08-2016",140084.599 
"03-09-2016",147221.2431
"03-10-2016",147230.5161
"03-11-2016",146403.4364
"03-12-2016",136844.3453
"03-13-2016",142078.372 
"03-14-2016",151942.6909
"03-15-2016",142223.9241
"03-16-2016",146177.8232
"03-17-2016",147613.9277
"03-18-2016",167238.0277
"03-19-2016",180384.4089
"03-20-2016",188590.7495
"03-21-2016",182810.843 
"03-22-2016",184224.1497
"03-23-2016",196359.3087
"03-24-2016",183263.1228
"03-25-2016",180812.2375
"03-26-2016",171958.1243
"03-27-2016",168369.2675
"03-28-2016",166556.3249
"03-29-2017",177034.76
"03-30-2016",170079.8884
"03-31-2016",138769.2261
"04-01-2016",104052.7466
"04-02-2016",101888.9868
"04-03-2016",97241.89454
"04-04-2016",113472.8918
"04-05-2016",113702.4615
"04-06-2016",114584.3604
"04-07-2016",139853.1619
"04-08-2016",141519.3142
"04-09-2016",133931.8941
"04-10-2016",134232.5185
"04-11-2016",148651.0414
"04-12-2016",155796.4331
"04-13-2016",166370.8387
"04-14-2016",176839.8545
"04-15-2016",162423.9259
"04-16-2016",159786.8232
"04-17-2016",158509.1189
"04-18-2016",193366.0834
"04-19-2016",189904.1701
"04-20-2016",203055.777 
"04-21-2017",198356.15
"04-22-2016",203288.3631
"04-23-2016",187005.0693
"04-24-2016",190706.6961
"04-25-2016",178835.4691
"04-26-2016",191424.1506
"04-27-2016",194468.083 
"04-28-2016",195130.4551
"04-29-2016",188871.3679
"04-30-2016",120729.2177
"05-01-2016",101883.54
"05-02-2016",132828.7452
"05-03-2016",140824.4341
"05-04-2016",144723.5316
"05-05-2016",144509.1919
"05-06-2016",148233.0282
"05-07-2016",132988.0482
"05-08-2016",155068.3301
"05-09-2016",177357.6171
"05-10-2016",150683.1777
"05-11-2016",156722.8054
"05-12-2016",161863.198 
"05-13-2016",163911.6095
"05-14-2016",153069.9453
"05-15-2016",156686.4311
"05-16-2016",166135.1524
"05-17-2016",174948.1918
"05-18-2016",169232.9176
"05-19-2016",171574.3804
"05-20-2016",180581.8887
"05-21-2016",176931.9823
"05-22-2016",177506.4972
"05-23-2016",205185.4856
"05-24-2016",199460.6449
"05-25-2016",201020.5857
"05-26-2016",199011.0097
"05-27-2016",207737.6986
"05-28-2016",186752.1881
"05-29-2016",178374.8064
"05-30-2016",170000.7001
"05-31-2016",169017.7576
"06-01-2016",130927.4144
"06-02-2016",161562.5926
"06-03-2016",166640.5323
"06-04-2016",154186.115 
"06-05-2016",164789.9911
"06-06-2016",183923.1578
"06-07-2016",163159.214 
"06-08-2016",166388.8603
"06-09-2016",174133.9206
"06-10-2016",178446.7731
"06-11-2016",162386.3313
"06-12-2016",156159.8653
"06-13-2016",183187.4152
"06-14-2016",172536.4929
"06-15-2016",190341.0299
"06-16-2016",196007.6102
"06-17-2016",208288.713 
"06-18-2016",194176.7152
"06-19-2016",190986.1126
"06-20-2016",203205.2892
"06-21-2016",196171.3974
"06-22-2016",204511.2539
"06-23-2016",207294.9062
"06-24-2016",210071.8986
"06-25-2016",191192.4362
"06-26-2016",188031.6754
"06-27-2016",217655.08
"06-28-2016",205813.6964
"06-29-2016",213295.5116
"06-30-2016",179185.5558
"07-01-2016",126970.6099
"07-02-2016",115723.302 
"07-03-2016",116296.3042
"07-04-2016",116432.6357
"07-05-2016",125613.8065
"07-06-2016",131694.6141
"07-07-2016",138072.3789
"07-08-2016",142148.7916
"07-09-2016",129954.6503
"07-10-2016",132408.2993
"07-11-2016",143657.812 
"07-12-2016",144839.8927
"07-13-2016",156252.4986
"07-14-2016",157180.5139
"07-15-2016",158413.5051
"07-16-2016",149903.7274
"07-17-2016",153065.3801
"07-18-2016",175308.8319
"07-19-2016",173443.3741
"07-20-2016",188558.3648
"07-21-2016",188707.344 
"07-22-2016",188178.5281
"07-23-2016",176515.1195
"07-24-2016",180302.697 
"07-25-2016",202685.3771
"07-26-2016",210364.3562
"07-27-2016",216578.2467
"07-28-2016",229242.5467
"07-29-2016",240760.0517
"07-30-2016",205097.2942
"07-31-2016",156741.1941
"08-01-2016",142755.2253
"08-02-2016",159156.3788
"08-03-2016",159863.1759
"08-04-2016",154890.6942
"08-05-2016",152813.3211
"08-06-2016",144691.4232
"08-07-2016",157967.6678
"08-08-2016",163263.452 
"08-09-2016",163589.9806
"08-10-2016",173830.4336
"08-11-2016",179155.6588
"08-12-2016",182260.8166
"08-13-2016",187212.0189
"08-14-2016",183644.0477
"08-15-2016",190392.4795
"08-16-2016",202435.6419
"08-17-2016",205086.9462
"08-18-2016",210798.5815
"08-19-2016",220526.1424
"08-20-2016",203264.3893
"08-21-2016",235486.1885
"08-22-2016",231055.1349
"08-23-2016",225934.6785
"08-24-2016",234770.9812
"08-25-2016",242472.2674
"08-26-2016",246040.0913
"08-27-2016",229852.4724
"08-28-2016",232808.4336
"08-29-2016",247352.1546
"08-30-2016",238722.3423
"08-31-2016",214732.8936
"09-01-2016",176471.5308
"09-02-2016",182245.8669
"09-03-2016",181619.0022
"09-04-2016",196978.6103
"09-05-2016",192220.0848
"09-06-2016",178061.1048
"09-07-2016",184735.3864
"09-08-2016",187288.3709
"09-09-2016",181908.8311
"09-10-2016",187910.307 
"09-11-2016",182545.4381
"09-12-2016",187514.9441
"09-13-2016",197155.3816
"09-14-2016",217742.4934
"09-15-2016",223862.5053
"09-16-2016",221006.1765
"09-17-2016",189872.4119
"09-18-2016",193843.169 
"09-19-2016",203675.9866
"09-20-2016",213811.8925
"09-21-2016",220208.7545
"09-22-2016",241377.8019
"09-23-2016",246041.8581
"09-24-2016",207586.0353
"09-25-2016",204159.0077
"09-26-2016",226439.5301
"09-27-2016",227182.1481
"09-28-2016",236468.0274
"09-29-2016",229575.3433
"09-30-2016",200519.5462
"10-01-2016",164300.1782
"10-02-2016",186050.4792
"10-03-2016",170200.7676
"10-04-2016",191515.8585
"10-05-2016",199026.0568
"10-06-2016",257863.7532
"10-07-2016",232799.3681
"10-08-2016",221876.2132
"10-09-2016",217059.6976
"10-10-2016",258061.0525
"10-11-2016",245271.929 
"10-12-2016",226812.5495
"10-13-2016",232548.6808
"10-14-2016",233217.6289
"10-15-2016",229611.1787
"10-16-2016",231719.5547
"10-17-2016",254768.9847
"10-18-2016",265116.2496
"10-19-2016",268885.4574
"10-20-2016",267400.6401
"10-21-2016",262910.5919
"10-22-2016",255859.7883
"10-23-2016",262778.4155
"10-24-2016",281951.3761
"10-25-2016",310437.2252
"10-26-2016",320165.7622
"10-27-2016",333842.451 
"10-28-2016",357960.7952
"10-29-2016",316344.3091
"10-30-2016",309959.7493
"10-31-2016",299658.7453
"11-01-2016",223897.4237
"11-02-2016",237513.5157
"11-03-2016",255539.9112
"11-04-2016",301579.1025
"11-05-2016",233583.1427
"11-06-2016",220690.6184
"11-07-2016",248397.2332
"11-08-2016",290195.0951
"11-09-2016",229567.1358
"11-10-2016",241020.9891
"11-11-2016",248818.2965
"11-12-2016",227226.7294
"11-13-2016",223216.2902
"11-14-2016",253030.7426
"11-15-2016",257570.0325
"11-16-2016",262019.9513
"11-17-2016",276079.8809
"11-18-2016",284187.5499
"11-19-2016",261298.1547
"11-20-2016",265156.1292
"11-21-2016",281307.1461
"11-22-2016",280176.8846
"11-23-2016",276438.5093
"11-24-2016",252663.8352
"11-25-2016",292407.5979
"11-26-2016",292563.6214
"11-27-2016",292683.1711
"11-28-2016",331890.1971
"11-29-2016",333568.3184
"11-30-2016",312808.6356
"12-01-2016",243918.7582
"12-02-2016",276656.5227
"12-03-2016",253026.7363
"12-04-2016",247064.4387
"12-05-2016",248566.0683
"12-06-2016",239281.7738
"12-07-2016",248347.0804
"12-08-2016",242702.3614
"12-09-2016",253755.8864
"12-10-2016",241270.3391
"12-11-2016",241292.9925
"12-12-2016",264475.8494
"12-13-2016",264406.9093
"12-14-2016",274134.9197
"12-15-2016",273249.6724
"12-16-2016",278228.6636
"12-17-2016",265495.9687
"12-18-2016",262624.5604
"12-19-2016",272875.3679
"12-20-2016",284914.0404
"12-21-2016",286892.1473
"12-22-2016",286563.0921
"12-23-2016",275578.1407
"12-24-2016",234701.9463
"12-25-2016",220856.0724
"12-26-2016",225274.8856
"12-27-2016",258263.7545
"12-28-2016",296016.1022
"12-29-2016",301813.6252
"12-30-2016",265173.9473
"12-31-2016",198707.5474
"01-01-2017",139089.5771
"01-02-2017",171617.2398
"01-03-2017",191143.2439
"01-04-2017",191398.6343
"01-05-2017",216797.6692
"01-06-2017",221755.9966
"01-07-2017",210156.7729
"01-08-2017",215025.4312
"01-09-2017",242135.5613
"01-10-2017",267748.3083
"01-11-2017",255192.1944
"01-12-2017",238464.0396
"01-13-2017",236009.9786
"01-14-2017",205253.6328
"01-15-2017",213225.3549
"01-16-2017",221689.2653
"01-17-2017",231543.6879
"01-18-2017",241793.0512
"01-19-2017",239296.4404
"01-20-2017",242145.3554
"01-21-2017",211035.3064
"01-22-2017",215715.1067
"01-23-2017",254076.8134
"01-24-2017",263862.481 
"01-25-2017",269089.7377
"01-26-2017",271923.4379
"01-27-2017",265442.1012
"01-28-2017",230757.0158
"01-29-2017",233561.859 
"01-30-2017",253732.8145
"01-31-2017",265151.455 
"02-01-2017",275807.2861
"02-02-2017",306178.7238
"02-03-2017",342401.7332
"02-04-2017",226133.0895
"02-05-2017",219946.564 
"02-06-2017",230291.5189
"02-07-2017",232565.3268
"02-08-2017",221004.4366
"02-09-2017",246777.2584
"02-10-2017",260837.0623
"02-11-2017",229799.3709
"02-12-2017",231210.2255
"02-13-2017",269810.1662
"02-14-2017",242573.9349
"02-15-2017",253060.5719
"02-16-2017",257771.6493
"02-17-2017",272021.9271
"02-18-2017",245059.9308
"02-19-2017",228844.9521
"02-20-2017",250230.675 
"02-21-2017",252631.5676
"02-22-2017",270649.1179
"02-23-2017",282819.3229
"02-24-2017",288309.6484
"02-25-2017",263492.9592
"02-26-2017",254797.3436
"02-27-2017",265253.3416
"02-28-2017",228427.2079
"03-01-2017",180671.241 
"03-02-2017",203298.6613
"03-03-2017",227114.9709
"03-04-2017",214727.3915
"03-05-2017",206483.2686
"03-06-2017",240510.6501
"03-07-2017",263735.1933
"03-08-2017",266391.6983
"03-09-2017",272892.7924
"03-10-2017",290161.5109
"03-11-2017",270168.4566
"03-12-2017",267177.5962
"03-13-2017",297388.7825
"03-14-2017",298096.8411
"03-15-2017",360933.1596
"03-16-2017",341256.9363
"03-17-2017",337883.1802
"03-18-2017",312312.3202
"03-19-2017",306722.699 
"03-20-2017",324767.047 
"03-21-2017",328337.1561
"03-22-2017",313745.1811
"03-23-2017",326987.6444
"03-24-2017",336442.9217
"03-25-2017",314248.3257
"03-26-2017",320622.3546
"03-27-2017",357928.0653
"03-28-2017",342982.1537
"03-29-2017",337220.4795
"03-30-2017",318551.2022
"03-31-2017",312178.5314
"04-01-2017",190402.066 
"04-02-2017",196899.5602
"04-03-2017",218774.7722
"04-04-2017",216423.3676
"04-05-2017",224985.4105
"04-06-2017",237777.1211
"04-07-2017",233741.8961
"04-08-2017",211684.6616
"04-09-2017",217522.1083
"04-10-2017",239418.286 
"04-11-2017",258127.749 
"04-12-2017",252168.3636
"04-13-2017",260779.9371
"04-14-2017",274653.6348
"04-15-2017",254062.4743
"04-16-2017",246196.4512
"04-17-2017",262100.4285
"04-18-2017",276273.6473
"04-19-2017",276256.5855
"04-20-2017",285040.6891
"04-21-2017",292438.5331
"04-22-2017",266195.6129
"04-23-2017",268645.9329
"04-24-2017",292025.7658
"04-25-2017",301491.6275
"04-26-2017",305571.151 
"04-27-2017",314408.4026
"04-28-2017",332090.0237
"04-29-2017",285591.1895
"04-30-2017",260793.1607
"05-01-2017",223645.5374
"05-02-2017",236130.6727
"05-03-2017",237112.6855
"05-04-2017",247173.3091
"05-05-2017",247360.9243
"05-06-2017",229966.4564
"05-07-2017",231775.2219
"05-08-2017",257436.0702
"05-09-2017",263334.1885
"05-10-2017",281250.2213
"05-11-2017",299357.6513
"05-12-2017",282819.397 
"05-13-2017",242172.3615
"05-14-2017",234316.5494
"05-15-2017",257881.4519
"05-16-2017",287765.8594
"05-17-2017",286259.2312
"05-18-2017",265744.2335
"05-19-2017",298025.507 
"05-20-2017",281333.8995
"05-21-2017",280527.8792
"05-22-2017",306235.8827
"05-23-2017",312357.6354
"05-24-2017",323433.5978
"05-25-2017",325469.4711
"05-26-2017",315888.1693
"05-27-2017",287133.2691
"05-28-2017",298943.6995
"05-29-2017",299363.003 
"05-30-2017",318668.0977
"05-31-2017",339687.0997
"06-01-2017",242207.861 
"06-02-2017",256342.19
"06-03-2017",241118.5469
"06-04-2017",251677.1928
"06-05-2017",300729.4038
"06-06-2017",308330.2909
"06-07-2017",311719.3691
"06-08-2017",306930.94
"06-09-2017",303488.1094
"06-10-2017",276151.7914
"06-11-2017",282285.8022
"06-12-2017",322342.8342
"06-13-2017",344140.8699
"06-14-2017",345999.8218
"06-15-2017",348117.8934
"06-16-2017",297688.5905
"06-17-2017",275951.1534
"06-18-2017",275273.8066
"06-19-2017",316915.532 
"06-20-2017",335618.3449
"06-21-2017",354567.1633
"06-22-2017",348296.5425
"06-23-2017",352294.075 
"06-24-2017",329673.0549
"06-25-2017",313638.378 
"06-26-2017",344946.2277
"06-27-2017",361493.619 
"06-28-2017",369594.5436
"06-29-2017",352461.3949
"06-30-2017",352924.3636
"07-01-2017",224606.2968
"07-02-2017",264562.1509
"07-03-2017",263579.7962
"07-04-2017",254388.9389
"07-05-2017",250895.845 
"07-06-2017",248566.2657
"07-07-2017",264419.5747
"07-08-2017",247071.5317
"07-09-2017",237234.725 
"07-10-2017",233508.5547
"07-11-2017",236736.3559
"07-12-2017",259391.155 
"07-13-2017",252122.6962
"07-14-2017",259074.6978
"07-15-2017",228874.1647
"07-16-2017",230126.3012
"07-17-2017",258937.9293
"07-18-2017",270348.2835
"07-19-2017",280504.7726
"07-20-2017",300120.4774
"07-21-2017",284508.1751
"07-22-2017",261362.0912
"07-23-2017",273726.9246
"07-24-2017",317566.027 
"07-25-2017",328644.3324
"07-26-2017",377204.6681
"07-27-2017",390080.9828
"07-28-2017",365884.1531
"07-29-2017",318684.3078
"07-30-2017",335563.5977
"07-31-2017",370429.4079

Answer 1

Whenever possible, it is best to develop one equation that effectively characterizes the data see “Joint estimation of all parameters is preferred.” from lecture 3 http://faculty.chicagobooth.edu/ruey.tsay/teaching/bs41202/sp2011/.

As you suggested, there are fairly strong deterministic factors in your data. We have seen this “problem/opportunity” while studying the demand for cash. Different days of the month, weeks-of-the-month, holiday effects (both lead and lag) etc. can have an important role. I took your 558 historical values and used AUTOBOX my tool of choice. This is the model that was automatically formed.

The statistics for this model are here

The Actual and Forecasts (next 81 days) are here

while the Actual, Fit and Forecast are shown here

A separate plot of the forecasts are here

with values here:

In summary there is a strong dependency on month-of-the-year, day-of-the-week (weekend effect) and week-of-the-month. In addition, AUTOBOX detected three time trends in the data and some unusual values. The timing of these pulses should/might be examined in order to suggest additional/omitted variables.

I consider this as an exercise in EDA (with apologies to Tukey) where the data is examined to flush out suggested assignable causes and a potentially useful model. In my long experience in dealing with daily data ARIMA structure is often an imperfect solution due to the fact that we are creatures of habit that often perform repeated functions based upon the hour, the day, the week and the month.

Finally, all models are wrong BUT some are useful (G.E.P. Box) the “BOX” in AUTOBOX. Hope this helps. If you wish to chat about the model/approach set up a chat room or contact me any way that you wish.

Answer 2

For this you will want to split the date into 3 parts (day, month, year) and then do a seasonal time series potentially. Chicaco booth have some good uni lecture notes available here (Try week 3): http://faculty.chicagobooth.edu/ruey.tsay/teaching/bs41202/sp2011/

R bloggers also have a brief section on this: https://www.r-bloggers.com/seasonal-or-periodic-time-series/