I have a variable, y, in 30 minute intervals, and is highly dependent both on the time of day and the day of the week - mainly Sundays. It is also dependent on bank holidays, which could by any day of the week.

I'm trying to fit a predictive model based on the data, using least squares regression.

I created a time series in R using the forecast package, with a weekly frequency, and created the fourier terms, around 20 seems to work well.

y_ts <- ts(dataframe$y, frequency = 48*7)
weekly_fourier <- forecast::fourier(y_ts, 20)
dataframe <- cbind(dataframe, weekly_fourier)

fit <- lm(y ~ .,dataframe) # code has been simplified

When I look at my fit, say for one month, I can see that the fit looks decent, and it has included the Sunday affect on the fit:


However the lower model prediction on Sundays is still very 'wiggly', I'd like it to be smoothed out. I tried selecting only fourier terms that had the most effect with a LASSO fit, but that just gave a small peak on Sundays, no flattening.

In addition, I'd like to try to factor in the bank holiday affect. I tried creating a bank holiday dummy variable as another feature in the training set, set as either 0 or 1, and trained on that, and the effect is noticeable (January 02 is a bank holday, left is without the dummy variable, right is with):

Without Bank Holiday With Bank Holiday

But it seems all it's done is shift the peak down rather than flatten.

My question is, what would be a good approach to take from here? Ultimately I'm trying to make a predictive model as accurate and sensible as possible, could I simply use heuristics and decide that for Sundays and Bank Holidays, the prediction should be a flat line independed on time of day? Or is there a more rigorous approach I can take.

  • $\begingroup$ A really quick comment on the wigglyness: jpeg works by throwing out the Fourier coefficients corresponding to high frequencies. For your case it might make sense to introduce a factor in front of the high freq terms being smaller on a Sunday and being 1 on every other day...(?) $\endgroup$ – Fabian Werner Nov 13 '18 at 14:08
  • $\begingroup$ Of course, this would just smooth the input signal to the model and does not help if you have a wiggly model on sundays... $\endgroup$ – Fabian Werner Nov 13 '18 at 14:28
  • $\begingroup$ That worked wonderfully. I had to set every fourier term equal to zero on sundays, then I just got the intercept. I can do the same on bank holidays. I can't upvote your comment sadly $\endgroup$ – zola25 Nov 13 '18 at 15:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.