Autocorrelation in timeseries

Question

I have daily measurements of a variable X in my timeseries. I was using two observations (D and D-1) to make forecast of a single day ahead (D+1). After plotting ACF, i got the following:

We can see that would be better to use D-7 and D-14, wich have higher correlation than D and D-1 (wich make sense to me. For example, let's say D is a monday, then D-7 represent the mondey a week earlier and D-14 represent a monday two weeks earlier). So, i used them as input in my model, but i actually got worse results in my forecast than i had when i was using D and D-1.

Besides using two days observation as input to my model, i also feed it with an one-hot-encode information of the day i wish to forecast. So, if i wish to forecast a monday, i would have an input similar to this:

X(D-14)| X(D-7)| Mon | Tue | Wed | Thur | Fri | Sat | Sun
10     | 20    | 1   | 0   | 0   | 0    | 0   | 0   | 0 

And for a Tuesday, would be something like this:

X(D-14)| X(D-7)| Mon | Tue | Wed | Thur | Fri | Sat | Sun
50     | 30    | 0   | 1   | 0   | 0    | 0   | 0   | 0 

So, does higher autocorrelated lagged values of a timeseries should always improve a model performance? If that's true, any ideas why my model actually got worse performance when making forecasts?

EDIT: My timeseries range from Jan/2017 until July/2022. For training, i am using the data from Jan/2017 until Dec/2021 (the last 20% of these data is used as validation set). So, for test, i'm using all months from Jan/2022 until July/2022. Iam using MAPE to assess my results. This MAPE is calculated per month. For example, i got these results when using D and D-1:

And, when using D-7 and D-14 i got these results:

Here is my timeseries plot, with the trend, seasonal and residual effect.

Just one more info, i am normalizing my timeseries between 0 and 1 (MinMaxScaler from Sklearn) before feeding it to train/test my model.

Answer 1

It is often hard to disentangle the effects of autocorrelation and seasonality.

Consider the case where your time series has only day-of-week seasonality, with no autocorrelative dynamics whatsoever. Essentially, you would be drawing random observations from seven different distributions, one for each day of the week. In your ACF plot, this would show up as peaks at lags 7, 14 etc. But capturing the seasonality through seasonal dummies would be the best you could do, and adding any autocorrelation to your model would be overfitting, because the true autocorrelation parameter, after accounting for seasonality, would be zero. So it is quite conceivable that your original model, with seasonal dummies and small autocorrelation lags, may perform better than one with higher order autocorrelations.

You could investigate this by fitting your data to your weekday dummies without autocorrelations, then plotting the ACF of the residuals. If regressing on weekdays already captures most of the effect, the peaks at lag 7, 14 etc. should mostly go away.

Also, try regressing on the weekdays only. The autocorrelation may not be helpful. You do have a trend, so you could try fitting a linearly increasing trend predictor. Also, you could try a standard AutoARIMA functionality with automatic model search, you definitely have enough data for that. It's not quite clear from your plots, but you may have multiple-seasonalities. There are specialized methods that can account for this. The tag wiki contains pointers to literature (and I would still try modeling the trend in addition).

Finally, you note you are using the MAPE. I assume your model does not use the MAPE in-sample as an optimization criterion, right? Also, it probably outputs expectation forecasts, not MAPE-minimal ones. Note that a MAPE-minimal forecast may be different from the expectation. However, with your large numbers, it likely does not make much of a difference (assuming you are calculating your MAPEs on the backtransformed data, not the 0-1 data - why are you scaling before fitting?). Still: What are the shortcomings of the Mean Absolute Percentage Error (MAPE)?

Answer 2

The first three autocorrelation coefficients in your time series are 0.7, 0.49, 0.41, which is just a little off from an AR(1) model with an autocorrelation of 0.7.

A seasonality of 7 days is also apparent from the peaks at 7, 14, 21 and 28 days.

The persistence of a significant positive autocorrelation out to 30 days and possibly beyond, is typical when a trend exists, but I do not believe this is your case, judging from the timeseries plot (top) which exhibits no trend. So it is possible that your data is better described via either integration (I1 or I2), or a long-range moving average (MA).

From the additional information in the comment, I gather that the trend plot is misleading as you fed the wrong value to the period parameter. While it is supposed to be 7 based on the autocorrelation, you fed it round(len(df) / 2), or half the length of the time series - so statsmodel was trying to find a periodicity of approximately 2.8 years in your data.

My suggestion is to correct the following in your analysis:

1. analyze the first difference of the Load column rather than the data in the column.
2. use period=7
3. Turn off trend extrapolation as there should not be any trend in your data.

In code, this translates to:

df['Load_d1'] = df['Load'].diff()
decomp = seasonal_decompose(df['Load_d1'], period=7, extrapolate_trend=0)

Check the standard deviations of your residuals:

np.std(decomp.resid)

The value of your residual standard deviation should be less than the standard deviation of the original data, as the seasonality and integration components should account for most of it.

At this point, you can use linear regression on the residuals, to find the AR(1) coefficient in the de-seasonalized data. Remember that the whole thing should be attempted on Load_d1 rather than Load.

To reconstruct the prediction of Load from the differentiated timeseries, use np.cumsum().