How to forecast daily time series with weekends and holidays?

Question

I'm having trouble choosing which approach to adopt when trying to forecast daily time series while taking into consideration special days like weekends and national holidays. The two methods I'm familiar with are :

Method 1 : using dummy variables to separate normal days from special days. Method 2 : separate normal days from special days and forecast each time series separately.

The first method seems the most intuitive and natural to me but it performs badly when the difference between normal days and special days is huge .. For instance when we have null values in the weekends. Furthermore, to implement this method in R I use the xreg attribute in ARIMA, but I don't know how to include dummy variables for other models like : ETS, STRUCTURAL, BATS & TBATS, THETA.

The second method seems too simple since it assumes that we have no relationship whatsoever between special days and normal days which is not the case in most cases.

Is there an other approach(s) I'm not aware of? If not, is there a way to improve those two methods? How can use dummy variables for models other than ARIMA in R?

Answer 1

Keep all data in one model: In my view, it is generally a bad idea to make separate models for subsets of your data. It is far preferable to create a single model that can adequately describe your entire data set. There are two reasons for this: (1) by creating separate models for subsets of the data, you increase the risk of over-fitting; and (2) by moving data out of the model, you reduce the information in both models, and therefore fail to estimate the parameters with all of the available data.

For that reason, in my opinion, the method of using dummy variables for weekends and holidays is far better than moving them to a separate model. You will need to decide how many dummies you want to use, which will require you to decide the level of detail (e.g., will both days of the weekend be considered the same?). Even with a sufficient number of dummy variables, you might sometimes encounter cases where the dummy variables are insufficient to model the data points well, and you might need interaction effects. If this is the case then you should consider adding more terms to improve the fit, but I would still recommend that you keep all the data in a single model.

Answer 2

You can use the rucm package in r to implement Unobserved Component Models which can accomodate dummies. I haven't used it but I am familiar with the methodology from using proc ucm in SAS.

I agree with you that it would be preferable not to split the data. Dummies may not work either though (and may not be necessary). If you set seasonality to a period of 7 (days in a week), then the dummies would be redundant with seasonality. You will likely be constrained to just one seasonal frequency. Would you be modeling monthly or week of year seasonality instead of day of week?

For special days, In my case I've had to deal with holidays (and the days surrounding holidays). I have set up a separate statistical procedure outside of the forecast that estimates the impact of each holiday, and the days around that holiday, and then I use it to adjust the forecast twice.

1. In advance of the holiday, when it's forecasted, make an adjustment to the prediction value and interval (let's say, a 25% increase).
2. After the holiday normalize the holiday to adjust its value back down (25% reduction) so that it does not cause an over-forecast.

Answer 3

https://autobox.com/capable.pdf ( SLIDE 47 ... ) will give you an example of what should be done ... essentially it is a version of your option 1 where lead and lag effects are identified and introduced along with factors like week-of-the=month , day-of-the-month , long weekends , dynamic causal input series, month-of-the-year , week-of-the-year ,day-of-the-week , Mondays-after holidays , Fridays-before holidays , level shifts , local time trends , changes in day-of-the-week patterns etc.

Answer 4

The difference between the two methods is irrelevant. With the same information as input, you should have the same information as output.

I tend to prefer the dummy variable approach because the degree of confidence is handled automatically by the software, which may soon becomes a nightmare if you use method 2 (especially with NULL observations).

The main exception is when it is not clear how to add dummy variables (or external variables in general) to the model. Which seams to be you case.