5
$\begingroup$

I have a time series with daily observations over the course of multiple years (interest in topic "superbowl" over time). The seasonality in the data is yearly as well and it is very spiky (almost nothing all year and big increase/spike in January/February). I have started using R for this task (forecast package) and have little experience with statistics.

x <- ts(myts, frequency=365)
fit <- HoltWinters(x)
plot(forecast(fit))

This works great and captures the seasonality of the data.

Now, I have read more about exponential smoothing (at http://otexts.com/fpp/7/) and understood that the HoltWinters model is one instance of the state space models implemented in ets. Unfortunately, I could not use ets so far since it complains about the high data frequency. I definitely need daily forecast (on the order of 30-60 steps).

fit <- ets(x, 'AAA')
Error in ets(x, "AAA") : Frequency too high

Why can HoltWinters deal with this but not ets? Is there a good workaround? I have the same problem for seasonal ARIMA models and considered splitting up the data in years and using past years as exogenous input.

On a side note: How do you usually deal with leap days that screw up your 365 day period? Simply delete them?

Thank you very much!

PS: I am aware of this: http://robjhyndman.com/researchtips/longseasonality/ However, I couldn't get it too work well on my data, yet. On the other hand, HoltWinters worked fairly well.

Thanks for all the helpful comments and discussion. I uploaded the data at http://timalthoff.de/data/data.zip The plot below shows Super_bowl.dat.

I took the liberty of including more time series if you'd like to check out more examples.

At certain points in time I want to forecast the time series on the order of 60 days. These points in time usually are on the left flank of a big spike that represents a sudden interest in a topic. See example.png for an example (the vertical red lines are these points in time to start an out-of-sample forecast). For more info check out the README.

enter image description here

Improve this question
$\endgroup$
6
  • $\begingroup$ This issue also comes up here. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Jan 22, 2013 at 15:21
  • $\begingroup$ Thanks Scortchi. If I understood correctly the only answer relates to the blog post linked in the PS. $\endgroup$
    – Tim
    Jan 22, 2013 at 15:43
  • $\begingroup$ Note that I uploaded the data as described in the comment to IrishStat's response. $\endgroup$
    – Tim
    Jan 23, 2013 at 10:57
  • $\begingroup$ Thanks for the data. Superbowl.dat is missing crucial information: what are the actual dates? This is important because we have valuable external information beforehand; namely, the exact scheduled date of the next several Super Bowls. Other information, such as day of week, might also be useful. (In light of one of your questions, we also would need to know where the leap day(s) occur.) Could you indicate exactly the range of dates covered in this file? $\endgroup$
    – whuber ♦
    Jan 23, 2013 at 16:52
  • $\begingroup$ I uploaded the dates: timalthoff.de/data/dates.dat The range is 2008/1/1 until 2012/12/31. These dates correspond to all time series in data.zip. There should be two leap days (2008 and 2012). It's okay to remove them if you like. $\endgroup$
    – Tim
    Jan 23, 2013 at 17:07

2 Answers 2

Reset to default
5
$\begingroup$

Try using the tbats() function in the forecast package for R:

x <- ts(myts, frequency=365)
fit <- tbats(x)
plot(forecast(fit))

TBATS is a generalization of ETS models designed to deal with high frequency data. See http://robjhyndman.com/papers/complex-seasonality/ for the JASA paper behind it.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Thanks! I tried it without and with seasonal.period=365 but it doesn't seem to capture the seasonality well. Also, longer forecast horizons seem to be a problem. Here are the forecast plots for TBATS, ETS, and HoltWinters (that seems to perform best right now). timalthoff.de/data/bats.jpg timalthoff.de/data/ets.jpg timalthoff.de/data/HoltWinters.jpg $\endgroup$
    – Tim
    Jan 23, 2013 at 10:52
  • $\begingroup$ The data clearly need to be logged first. $\endgroup$
    – Rob Hyndman
    Jan 24, 2013 at 0:53
  • $\begingroup$ Your bats model and ets model have no seasonality at all. You can't expect them to work if you don't specify the seasonality. $\endgroup$
    – Rob Hyndman
    Jan 24, 2013 at 1:17
4
$\begingroup$

R may be of little use to you due to the complexity of your problem. We recently developed forecasts for a daily series that looks "similar" but "different" to yours.

enter image description here .

It required combining Intervention Detection , Causal Variables (Holidays , Promotions etc.) and ARIMA structure. We used AUTOBOX ( a product that I have helped develop ) to do this. You can post your data on the board and I will post some results. This is indeed one of the most challenging time series that we have seen ..... and we have been looking for a long time ! It might be interesting to see how other thought leaders would analyze these data and compare the results.

EDITED After receiving the data from the OP

Data like this requires incorporation of Fixed Events i.e. Monthly/Daily/Holidays along with specially formed days-of-year where significant activity can be detected around the event. The OP posted data and asked for forecasts 60 periods precisely prior to the Super Bowl. He delivered 1827 daily values starting at 1/1/12008. There were three origins for the forecast. Only data up to the point of the origin was used to develop the model/parameters. The three origins were 12/9/2010 (1074 values used), 12/8/2011 (1438 values used) and 12/5/2012(1801 values used). The three Actual-Fit-Forecast Graphs are presented here.First with 1065 values enter image description here Then with 1438 values enter image description here and finally using all the data (1801 values) enter image description here

The complete analyses can be found at

http://www.autobox.com/1074.zip

http://www.autobox.com/1438.zip

http://www.autobox.com/1801.zip

Each file contains an xls/xlsx file containing the 60 forecasts and other files showing all the analysis. Each equation is different because the number of observations used to identify the model changed. Following is the equation used from period 12/5/2012 to predict the 60 days leading up to Super Bowl Sunday. The equation uses daily indicators reflecting the buildup before Super Bowl Sunday

M_SB is a 0/1 variable denoting the day for the Super Bowl , while M_1DB through M_22DB are the days before Super Bowl Sunday and M_1DA through M_3DA are for the days after Super Bowl. In addition there significant responses around 4 other holidays. Jan and Feb have a significant impact along with 6 daily indicators (N10107 through N10607). A significant ARIMA structure was found to round out the model.
Y(T) = -12757.
+[X1(T)][(+ 20512. )] M_SB

   +[X2(T)][(+  8680.2    )]                           M_1DB

   +[X3(T)][(+  1688.0    )]                           M_2DB

   +[X4(T)][(+  2778.1    )]                           M_3DB

   +[X5(T)][(+  1906.4    )]                           M_4DB

   +[X6(T)][(+  1222.0    )]                           M_5DB

   +[X7(T)][(+  829.06    )]                           M_6DB

   +[X8(T)][(+  948.29    )]                           M_7DB

   +[X9(T)][(+  397.93    )]                           M_8DB

   +[X10(T)[(+  509.42    )]                           M_10DF

   +[X11(T)[(+  804.90    )]                           M_11DB

   +[X12(T)[(+  1102.0    )]                           M_12DB

   +[X13(T)[(+  1867.1    )]                           M_13DB

   +[X14(T)[(+  10258.    )]                           M_14DB

   +[X15(T)[(+  754.71    )]                           M_15DB

   +[X16(T)[(+  328.09    )]                           M_17DB

   +[X17(T)[(+  10116.    )]                           M_21DB

   +[X18(T)[(+  1467.6    )]                           M_22DB

   +[X19(T)[(+  1113.0    )]                           M_1DA

   +[X20(T)[(-  673.57    )]                           M_2DA

   +[X21(T)[(+  601.89    )]                           M_3DA

   +[X22(T)[(+  584.44    B**-2+  1669.4    B**-1+  808.45 +  345.02    B** 1)]       M_MARDIGRAS
   +[X23(T)[(-  7812.7    )]                           M_MARTINLKING

   +[X24(T)[(-  541.22    )]                           M_NEWYEARS

   +[X25(T)[(-  529.21    -  389.18    B** 1)]         M_PRESIDENTS

   +[X26(T)[(+  705.02    )]                           MONTH_EFF01

   +[X27(T)[(+  605.10    )]                           MONTH_EFF02

   +[X28(T)[(+  13116.    )]                           FIXED_EFF_N10107

   +[X29(T)[(+  13017.    )]                           FIXED_EFF_N10207

   +[X30(T)[(+  12971.    )]                           FIXED_EFF_N10307

   +[X31(T)[(+  12974.    )]                           FIXED_EFF_N10407

   +[X32(T)[(+  12917.    )]                           FIXED_EFF_N10507

   +[X33(T)[(+  13036.    )]                           FIXED_EFF_N10607

         +     [(1-  .626B** 1)(1-  .249B** 7)]**-1  [A(T)]
Improve this answer
$\endgroup$
8
  • $\begingroup$ Tim, if you want to take IrishStat up on this offer--and it's real, by the way, as many of his other answers here will attest--then I recommend not posting the last full year of your data. Hold on to that and use it to check the accuracy of any forecasts made from the rest of the data. BTW, a good way to post a large amount of data is to make it available somewhere else on the Web in a generic format, such as a text file, and provide a link. $\endgroup$
    – whuber ♦
    Jan 22, 2013 at 21:26
  • $\begingroup$ @whuber There is a problem with assessing accuracy from a single origin as the NF forecasts are often autocorrelated, thus one could call this a sample of 1. If an additional k values are then released to the model an updated set of forecasts can then be made (a second sample of NF forecasts). This can be done from K origins. Essentially we will then have K (# of origins) sets of NF forecasts and then one can compute error measures for different lead times up to NF and compare procedures/methods . One swallow does not a summer make comes to mind. $\endgroup$
    – IrishStat
    Jan 22, 2013 at 21:45
  • $\begingroup$ If some of the data are not held out, then it would be impossible to conduct a fair or objective predictive trial. I don't much care how you assess the accuracy of your forecasts (although the OP ought to care), but it has to be done by comparison to future data that were unavailable to you at the time of model creation and fitting. (Holding out past data, BTW, strikes me as being a rather useless test.) As your know, the need to check models through independent results is a general scientific and statistical principle; I don't see why time series data should be exempt from it. $\endgroup$
    – whuber ♦
    Jan 22, 2013 at 21:53
  • 1
    $\begingroup$ It is not exempt. What I was trying to suggest that one would withhold data ( as you suggest ) aand then make forecasts from a number of origins. The withheld data would never be used to alter the model or the paramers as that would be "cheating" . $\endgroup$
    – IrishStat
    Jan 22, 2013 at 22:11
  • $\begingroup$ Thank you very much for that analysis! Unfortunately, I get a 404 error when accessing the files you provided. Generally, I will not be able to use a variable for the date of the super bowl as I am looking at events that possibly have no fixed date or where that date is hard to determine in a fully automatic fashion. I have the following questions regarding your analysis and findings (see next comment due to space restriction): $\endgroup$
    – Tim
    Jan 25, 2013 at 9:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.