10
$\begingroup$

I'm having trouble choosing which approach to adopt when trying to forecast daily time series while taking into consideration special days like weekends and national holidays. The two methods I'm familiar with are :

Method 1 : using dummy variables to separate normal days from special days. Method 2 : separate normal days from special days and forecast each time series separately.

The first method seems the most intuitive and natural to me but it performs badly when the difference between normal days and special days is huge .. For instance when we have null values in the weekends. Furthermore, to implement this method in R I use the xreg attribute in ARIMA, but I don't know how to include dummy variables for other models like : ETS, STRUCTURAL, BATS & TBATS, THETA.

The second method seems too simple since it assumes that we have no relationship whatsoever between special days and normal days which is not the case in most cases.

Is there an other approach(s) I'm not aware of? If not, is there a way to improve those two methods? How can use dummy variables for models other than ARIMA in R?

$\endgroup$
1

4 Answers 4

7
$\begingroup$

Keep all data in one model: In my view, it is generally a bad idea to make separate models for subsets of your data. It is far preferable to create a single model that can adequately describe your entire data set. There are two reasons for this: (1) by creating separate models for subsets of the data, you increase the risk of over-fitting; and (2) by moving data out of the model, you reduce the information in both models, and therefore fail to estimate the parameters with all of the available data.

For that reason, in my opinion, the method of using dummy variables for weekends and holidays is far better than moving them to a separate model. You will need to decide how many dummies you want to use, which will require you to decide the level of detail (e.g., will both days of the weekend be considered the same?). Even with a sufficient number of dummy variables, you might sometimes encounter cases where the dummy variables are insufficient to model the data points well, and you might need interaction effects. If this is the case then you should consider adding more terms to improve the fit, but I would still recommend that you keep all the data in a single model.

$\endgroup$
1
  • 1
    $\begingroup$ in the case where modelling subsets separately gives good results, using interactions should also perform well; then using a regulariser will ensure common behaviour between subsets is estimated on the main variables. $\endgroup$
    – seanv507
    Mar 10, 2019 at 18:16
0
$\begingroup$

You can use the rucm package in r to implement Unobserved Component Models which can accomodate dummies. I haven't used it but I am familiar with the methodology from using proc ucm in SAS.

I agree with you that it would be preferable not to split the data. Dummies may not work either though (and may not be necessary). If you set seasonality to a period of 7 (days in a week), then the dummies would be redundant with seasonality. You will likely be constrained to just one seasonal frequency. Would you be modeling monthly or week of year seasonality instead of day of week?

For special days, In my case I've had to deal with holidays (and the days surrounding holidays). I have set up a separate statistical procedure outside of the forecast that estimates the impact of each holiday, and the days around that holiday, and then I use it to adjust the forecast twice.

  1. In advance of the holiday, when it's forecasted, make an adjustment to the prediction value and interval (let's say, a 25% increase).
  2. After the holiday normalize the holiday to adjust its value back down (25% reduction) so that it does not cause an over-forecast.
$\endgroup$
1
  • 1
    $\begingroup$ Thanks for your suggestion to use the "rucm" package, the problem is I need to use those specific models I mentioned in my question. I like your way of handling special days though, it gives you more control on the effect of holidays. In regards to your question, I work with multi seasonal time series : daily observations with weekly and annual seasonality. $\endgroup$
    – Taha
    Aug 21, 2018 at 17:08
0
$\begingroup$

https://autobox.com/capable.pdf ( SLIDE 47 ... ) will give you an example of what should be done ... essentially it is a version of your option 1 where lead and lag effects are identified and introduced along with factors like week-of-the=month , day-of-the-month , long weekends , dynamic causal input series, month-of-the-year , week-of-the-year ,day-of-the-week , Mondays-after holidays , Fridays-before holidays , level shifts , local time trends , changes in day-of-the-week patterns etc.

$\endgroup$
-1
$\begingroup$

The difference between the two methods is irrelevant. With the same information as input, you should have the same information as output.

I tend to prefer the dummy variable approach because the degree of confidence is handled automatically by the software, which may soon becomes a nightmare if you use method 2 (especially with NULL observations).

The main exception is when it is not clear how to add dummy variables (or external variables in general) to the model. Which seams to be you case.

$\endgroup$
5
  • 1
    $\begingroup$ That's what I thought before trying to forecast a daily time series with null values using ARIMA and a dummy variable to separate weekends from normal days; the results I got suggests that the two methods I mentioned are different since I got none null forecasted values for the future weekends! $\endgroup$
    – Taha
    Aug 21, 2018 at 17:14
  • $\begingroup$ Strange. And mathematically baffling, because the dummy variable absorbs all the difference. Maybe you have 3 dummy variables (Saturday, Sunday , Bank Holidays). $\endgroup$
    – AlainD
    Aug 22, 2018 at 6:25
  • $\begingroup$ It would be interesting if you could publish a graph and/or some data. Possibly after having done a linear transformation to keep your data private. $\endgroup$
    – AlainD
    Aug 22, 2018 at 6:27
  • $\begingroup$ okey I'll publish the procedure and the results I got as soon as possible . $\endgroup$
    – Taha
    Aug 22, 2018 at 12:55
  • $\begingroup$ -1 for this opening: "The difference between the two methods is irrelevant. With the same information as input, you should have the same information as output." This baffling statement would imply that it never matters what statistical procedure one uses, because the output would be the same. $\endgroup$
    – mkt
    Jul 20, 2022 at 15:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.