I have a variable, y, in 30 minute intervals, and is highly dependent both on the time of day and the day of the week - mainly Sundays. It is also dependent on bank holidays, which could by any day of the week.
I'm trying to fit a predictive model based on the data, using least squares regression.
I created a time series in R using the forecast package, with a weekly frequency, and created the fourier terms, around 20 seems to work well.
y_ts <- ts(dataframe$y, frequency = 48*7) weekly_fourier <- forecast::fourier(y_ts, 20) dataframe <- cbind(dataframe, weekly_fourier) fit <- lm(y ~ .,dataframe) # code has been simplified
When I look at my fit, say for one month, I can see that the fit looks decent, and it has included the Sunday affect on the fit:
However the lower model prediction on Sundays is still very 'wiggly', I'd like it to be smoothed out. I tried selecting only fourier terms that had the most effect with a LASSO fit, but that just gave a small peak on Sundays, no flattening.
In addition, I'd like to try to factor in the bank holiday affect. I tried creating a bank holiday dummy variable as another feature in the training set, set as either 0 or 1, and trained on that, and the effect is noticeable (January 02 is a bank holday, left is without the dummy variable, right is with):
But it seems all it's done is shift the peak down rather than flatten.
My question is, what would be a good approach to take from here? Ultimately I'm trying to make a predictive model as accurate and sensible as possible, could I simply use heuristics and decide that for Sundays and Bank Holidays, the prediction should be a flat line independed on time of day? Or is there a more rigorous approach I can take.