# How do I compare time series forecast models? (ARIMA vs HoltWinter)

I'm working on a toy problem to try and get a better understanding for time series forecasting. I have a sample data set, which I'll include, that shows daily e-commerce sales from 2015 through Feb, 2017.

I tried a arima model and holtwinter and they both show up as significant against my box test and they have residuals that are roughly normal and centered on zero.

But they give pretty wildly different predictions for the future. I'm curious if there are more ways I can evaluate to find out which model is correct. I'll include my data, my code and the forecasts.

### Data

print(ts)
Time Series:
Start = c(2015, 1)
End = c(2017, 54)
Frequency = 365
[1]  2419.79487  3505.31069  3639.80925  1612.65730  2422.11190
[6]  6313.53477  5356.30223  1948.21457  1226.27511  2619.86945
[11]  1356.79885  2615.20348   912.33441  2000.43040  1508.11609
[16]   931.46061   122.07641   330.40875   834.80850  2237.23233
[21]  1510.53502   665.06950  6863.78618   639.22052    89.02246
[26]  2754.30644   268.26101  3697.11548  1267.50463  2372.73719
[31]   829.36903  1824.97771  2194.93701  3646.11185  3759.12441
[36]  2525.68906  2331.31828   890.27223  1120.82170  1805.74774
[41]   972.69618  2447.54541  3081.15920  1488.90617   147.64338
[46]  4340.27688  2176.31350  4411.59523  1792.95687  3609.35958
[51]  2286.18995  2222.49398  3593.66190  3900.08497  1892.80978
[56]  2665.22172  4276.53924    50.44121  4067.94254  2628.02807
[61]   541.83367  1117.90369  3015.66985  1016.28566  2537.17051
[66]   516.67809  6386.00200   418.82217  2949.50880  1689.55913
[71]  2949.91583  2726.65076  2884.76752  2229.08619    48.13187
[76]  3326.91954  2956.76084  2010.63298   871.95161   496.76691
[81]  2921.75271  1086.45719  1281.92972   846.14622   964.69261
[86]  2931.91579  2793.25232   719.45558   400.91642 10346.59135
[91]  1023.74518  2013.82867  2817.03580  4265.75481  2620.27636
[96]   900.70305  5631.09486   251.03763  3102.22359  2581.32491
[101]  5568.12748  2580.34334  3528.18579  6455.47190  2508.56013
[106]  3205.39106  1925.11516  1016.13844  1186.39525  3516.92571
[111]  1379.72208  2398.55234  2803.17791   647.70648  4692.30235
[116]  3368.73469  1883.75659  9133.57446  1812.21848  1564.08438
[121]  4227.42480  8565.40062  1924.97101  2223.61876  4181.95294
[126]  1977.34741  3760.25765  2499.33240  5044.08621  4765.99922
[131]  4041.15915  3821.23957  2415.37833  1962.89834  7403.01233
[136]  3019.91717   305.22969  4845.42196   445.80997  3610.68729
[141]  1383.78085  2767.74932  1734.36058  3292.70695  6547.25405
[146]  2513.03910  5560.55729  2716.36603  6231.96621  1500.73467
[151]  1497.32254  3266.17682  2870.74022  1681.59394  1530.76419
[156]  3104.80550  4872.74537  2815.93168  6455.70635  6545.21991
[161]  5092.66683  8178.01542 12568.84644  6322.05778  1765.43631
[166]  6626.02960  1668.03191  3914.24293  4566.22668  8365.60121
[171]  3099.42183  5446.67106  3200.00943  1224.58919  1559.96960
[176]  2138.57382  1906.66929  2422.35368  3616.86548  3334.00640
[181]  4507.14149  6125.92448  8092.22990  3794.75678  2865.56637
[186]  2227.67440  5956.54937  4588.17040  2043.45609  5434.65938
[191]  6875.76103  2305.23861  4346.69014  1597.73418  1940.89045
[196]  4257.32500  4772.42652  3589.84997  5067.06486  4945.17905
[201]  6301.70996  4148.84441  3657.72890  2864.09213  9503.47545
[206]  1255.75135  1748.96309  6303.93657  3382.48993  5573.37107
[211]  5371.02390  7488.83602  6247.43076  1436.65204  2765.54182
[216]  4784.58157  2853.47190  6896.63376  7417.30945  3119.94501
[221]  5465.63318  9422.17774  2417.14581  4929.78458  3495.64228
[226] 10153.46685  1187.32065  4873.73448  8732.30121  1587.15222
[231]  4601.50141  7653.46015  4663.40680  4144.93244  2475.20900
[236]  8588.03985  3347.64382  6091.36366  3314.00860  4044.20706
[241]  2097.46376  6283.00718  8423.43238  3744.23710  6027.19343
[246]  6336.40387  9338.00781  7554.75618  2756.94885  5186.63034
[251]  2421.60522  6987.49500  1597.06331  6720.19045  2744.80513
[256]  3038.76154  8178.66984  3926.00886  5940.65434  5300.48768
[261]  8471.32821  8928.95587  4701.39813 10041.44358  1680.14400
[266]  3968.45973  4472.53240 13293.86963  3699.19469  1467.66668
[271]  2461.11422  5750.48134  1414.33112 11885.92798  6386.73945
[276]  3378.33576  1713.67842  8359.66591  4028.41458  1744.99820
[281]  5981.58892 11398.67841  7566.50843  3817.43666  4632.71624
[286]  5901.94714  4984.39088  8637.71053  6499.54514  4092.10800
[291]  3251.86070  9449.60580  1854.83255  3778.81510  1893.59039
[296]  6996.37762  2710.17601 11662.40925 11698.90749  3855.53297
[301]  3771.92696  7342.43176  5014.54362 10157.92248  6429.03165
[306]  3454.77741  3460.79925  5958.29986  1866.37881  5927.29513
[311]  3647.21311  3617.65112  8250.16291  5369.64612  5961.59530
[316]  6740.12253 11509.91538  7687.55905  3289.38129 10515.51580
[321]  2819.35782  8545.49493  4219.39841  2180.53320  5634.11421
[326]  2980.08294  8943.12231  5906.84697  3458.74814  5231.12158
[331] 10244.34633  6439.00793  4403.12888  7397.66225  5676.87007
[336]  6051.90782  6847.33061  4331.30326  9647.08626  7596.30480
[341] 11282.27388  3605.06924  3887.58673 10847.82180  2404.46687
[346]  5898.25409  7008.69696  5065.71015  4244.98757  5000.98470
[351]  4512.22093 18559.29707  6126.12144  3362.73204  9170.10520
[356]  4991.86323 11226.08310  3967.26723  9927.51505  3202.16174
[361]  9744.17375 17283.79430  4992.50816  5511.12493  6089.36367
[366] 13661.67186  4781.35550  9060.74923  7968.16525  4301.33955
[371]  7915.49338  4051.91004 15439.58373  2625.92301  4636.23597
[376] 17518.70264  1489.57577  3752.88701  7186.81737 18418.39117
[381]  5215.34041 14320.49173 10836.93392  7122.68583  3827.34641
[386]  9658.48586 11540.33199  4874.93156  5413.79996  6559.87721
[391]  6698.35115  7450.31618  5124.40833 11848.79921 10472.24870
[396]  8947.89722  8322.23170  5562.67212  6697.50212 10278.24219
[401] 19850.43405  5668.77950  6669.72378 13966.87880  7848.34134
[406] 10852.47518 14119.84491 19456.65210  7549.29552  7646.82791
[411] 13858.53778  4870.83920  7321.03738 10130.07088 22499.67895
[416] 12859.81724  4525.16104 13087.18152  4211.61067  5297.96578
[421]  9758.97568 13191.58188 10540.24401  6805.64653 11401.71238
[426]  4674.43318  4894.43519  4595.99790 18100.55951  6155.16499
[431]  7537.91264 10502.84843  7197.80368  4492.28798 10915.47578
[436] 12137.70002  6246.59295 10366.29153 19848.90448  4425.16107
[441]  6414.59828 18595.67532 14526.65009  3581.70104 10361.29231
[446] 23150.30732  1629.64036  7787.46025  7666.88010 14235.82678
[451]  7035.54010 11977.38129 11801.51601 12376.49013  5831.76278
[456]  9898.75818 16525.40683  5768.75332 10566.59566 24055.29488
[461] 10073.53801  9009.13923  9319.13438 13907.98684 10925.22464
[466]  9792.87148 16977.72332  3774.61189  8413.25801  9750.37538
[471] 28654.80504  7348.77271  8303.36030 12211.30824  9790.53693
[476]  9601.36392  9825.83614 22485.91512 11552.05697  5450.97418
[481] 19013.68849  7399.47044 10804.38369  6607.49470  8698.30219
[486]  9045.30062 14120.25877 13995.96400 12991.81683  8942.38113
[491]  6139.69811 10048.96559  9713.64216  8688.18051 27664.79188
[496]  8692.95157  6988.05436 12573.54472 13313.57444 10093.68181
[501]  3542.07225 24604.21248 10821.76302  5245.51342 11803.05721
[506]  9626.88337 13010.85279 16794.29634 16541.29423  7811.71742
[511] 11497.11999  9490.90880 15691.86496 11322.97888 12059.74836
[516] 29516.08831  8788.84032 10553.92136 13731.43889 16354.18013
[521] 17382.61772 10376.21927 21303.19616  6331.06522  6978.52080
[526]  9406.21650 21248.21329 19079.57078 16643.10817 14178.88970
[531]  9777.38379 10307.55443  9122.59614 23274.43110 17496.65486
[536]  7988.07085 19218.40662 10822.65428 10373.89506 10572.60838
[541] 19175.31931 18464.68286 11610.55153 26854.05031  9342.41123
[546] 14780.56518 18678.74688 12739.95370 11753.04279  5443.18343
[551] 27418.34257 10140.25222 15708.84392 17922.23717 25052.15779
[556] 13128.82365  9235.30723 23451.62767 10084.03463 10317.88871
[561] 14292.19289 27673.55812 10063.59796 12965.26029  9704.30161
[566]  9460.62887 11391.32989 10939.61661 22451.54530  9323.79815
[571] 15173.53856 16338.60835  8907.35576 10319.21918 15804.69749
[576] 22248.90622 10582.54606 11571.06553 22203.13622 13314.77408
[581] 20050.21434 15310.16295 13890.42942 11648.31834  6440.00580
[586] 18626.58934  6865.56176 10249.83446 16047.41271 21852.21472
[591] 22609.51047  9338.29901 12546.99784  7509.13902 11543.12718
[596] 14926.37920 29941.14068 16301.91987 13267.19478 20968.55969
[601]  9888.97530 11325.35852 14439.56179 31678.69639 15126.80778
[606] 11677.04261 19254.75438 12989.73702 14566.32013 15678.70193
[611] 14385.02343 12792.98629 16176.70584 33500.16830 13180.92931
[616] 16400.46322 29907.77121 21393.57842 15766.37736 12175.14383
[621] 19742.19402 12480.24253  9667.78266 19113.82963 27009.60011
[626] 17181.73216 12862.94088 20154.16806  8918.23393 10656.65783
[631] 15354.68490 35984.08773 21859.95243 12571.96797 24392.90865
[636] 13523.15454 17531.80812 17107.54903 16011.70841 19153.31569
[641] 13507.97139 24859.11373 12160.92744  7658.77275 12652.25339
[646] 20786.80630 19219.05594 14001.75736 23930.09355 14781.50495
[651] 16768.75157 17523.10956 27227.61232 16986.16377 11866.47244
[656] 32843.62019 10429.48463 19272.35178 32215.45883 25017.83671
[661] 11263.33090 16578.51746 35580.88951 14912.92940 16244.67581
[666] 19365.00181 30946.50694 18750.29772 17461.82259 24688.60349
[671] 13733.52349 11490.22267 22605.09096 19785.42353 19251.76745
[676] 15444.40320 26905.76718 11515.96307 17475.84202 20306.29457
[681] 26201.40881 23415.37499 13463.03016 29751.80143 14979.88358
[686] 19665.15285 24472.71512 33866.71294 23968.12308 14954.76833
[691] 36554.00655 19123.46527 15219.26998 33079.20749 23262.99565
[696] 19912.45876 13201.47413 41280.95619 15334.89906 21605.91294
[701] 16903.39509 37435.43722 21108.17089 23374.52077 35857.52232
[706] 21393.97474 21250.53593 24892.08621 24498.68640 18589.12106
[711] 19965.10975 36869.90623 12690.09279 14958.43143 21128.64867
[716] 28153.35056 15406.95845 15033.53285 43978.06743 15926.07385
[721] 17957.29328 27003.36373 32398.15296 26932.84454 19612.79280
[726] 27972.24553 13460.94492 15299.55838 23859.95146 26053.24987
[731] 18806.59707 19577.55827 51509.19952 16139.87143 19866.36628
[736] 18413.60302 32198.78960 21738.99945 18004.37841 38202.37585
[741] 15216.14492 18578.28043 26806.46957 30772.19407 18713.38384
[746] 18692.23514 41750.56528 14110.49392 16397.12035 19966.07880
[751] 26811.19010 22338.63547 22835.77244 30527.05657 13049.25827
[756] 12460.07663 30202.84367 30332.39829 24582.66513 16044.06816
[761] 36128.65052 19173.09364 17540.32332 15495.95355 39799.96467
[766] 18238.17887 19218.93387 40014.99026 14646.82376 18970.33616
[771] 29382.22482 28078.51323 24561.64152 20083.82957 38732.65776
[776] 18797.43935 18347.29107 23013.29072 19419.47252 18180.54027
[781] 17270.98478 41243.08957 17319.42314 17030.13515


### Code

library(forecast)
# model
# arima
find_arima <- auto.arima(ts, stepwise = FALSE, approximation = FALSE)
fit <- arima(ts, c(4, 1, 1))
plot(forecast(find_arima, h = 365))

# holtwinter
hw <- HoltWinters(ts)
hw_forecast <- forecast(hw, h = 365)

# evaluate
# arima
acf(find_arima$$residuals, lag.max=200) Box.test(find_arima$$residuals, lag=200, type="Ljung-Box")
plot(find_arima\$residuals)

# holtwinter
acf(hw_forecast, lag.max=20)
Box.test(hw_forecast$$residuals, lag=200, type="Ljung-Box") plot(hw_forecast$$residuals)


### Forecasts

Next Year Forecasts with ARIMA

Next Year Forecasts with HoltWinter

ARIMA shows a consistent positive trend for the next year while HoltWinter indicates it could go in either direction, but it will probably be negative.

I know in a regular model I can build a test set and compare against that, but I don't know if that holds true with time series models. Is there a definitive way to find out which model is better?

I split my data into train and test and this is the forecast with the lowest RMSE.

• I think the answer to all of your problems is a test set! May 27, 2019 at 8:40
• These are all great answers and I'll include my finished forecast because it's pretty cool.
– ivan
May 29, 2019 at 1:33