2
$\begingroup$

I have daily measurements of a variable X in my timeseries. I was using two observations (D and D-1) to make forecast of a single day ahead (D+1). After plotting ACF, i got the following:

enter image description here

We can see that would be better to use D-7 and D-14, wich have higher correlation than D and D-1 (wich make sense to me. For example, let's say D is a monday, then D-7 represent the mondey a week earlier and D-14 represent a monday two weeks earlier). So, i used them as input in my model, but i actually got worse results in my forecast than i had when i was using D and D-1.

Besides using two days observation as input to my model, i also feed it with an one-hot-encode information of the day i wish to forecast. So, if i wish to forecast a monday, i would have an input similar to this:

X(D-14)| X(D-7)| Mon | Tue | Wed | Thur | Fri | Sat | Sun
10     | 20    | 1   | 0   | 0   | 0    | 0   | 0   | 0 

And for a Tuesday, would be something like this:

X(D-14)| X(D-7)| Mon | Tue | Wed | Thur | Fri | Sat | Sun
50     | 30    | 0   | 1   | 0   | 0    | 0   | 0   | 0 

So, does higher autocorrelated lagged values of a timeseries should always improve a model performance? If that's true, any ideas why my model actually got worse performance when making forecasts?

EDIT: My timeseries range from Jan/2017 until July/2022. For training, i am using the data from Jan/2017 until Dec/2021 (the last 20% of these data is used as validation set). So, for test, i'm using all months from Jan/2022 until July/2022. Iam using MAPE to assess my results. This MAPE is calculated per month. For example, i got these results when using D and D-1:

enter image description here

And, when using D-7 and D-14 i got these results:

enter image description here

Here is my timeseries plot, with the trend, seasonal and residual effect.

enter image description here

Just one more info, i am normalizing my timeseries between 0 and 1 (MinMaxScaler from Sklearn) before feeding it to train/test my model.

$\endgroup$
4
  • $\begingroup$ How do you assess your results? What accuracy measure are you using? Are you using a holdout sample? If so, how long is it, and how long is your time series? What kinds of measurements are these? Can you include a plot of your series? $\endgroup$ Commented Oct 26, 2022 at 22:14
  • $\begingroup$ @StephanKolassa i have edited the post with some more information you aksed. $\endgroup$
    – Murilo
    Commented Oct 26, 2022 at 23:18
  • $\begingroup$ How did you get the trend plot (the one labeled tendência)? It's looks like some mixture of linear regression (before 29/05/2018 and after 20/03/21) and long-range moving average. Is there some heuristic in the background? How did you get the seasonality plot? What period of seasonality does it represent? How come it is neither purely cyclical nor completely trend-free? For a series with a trend, the common practice is to work with the first difference, rather than the original series. $\endgroup$
    – KishKash
    Commented Oct 27, 2022 at 7:51
  • $\begingroup$ @KishKash Sorry for the portuguese labels. But everything was plotted with seasonal_decompose from the statsmodel Python module. I just did the following: decomp = seasonal_decompose(df['Load'], period=round(len(df) / 2), extrapolate_trend='freq') and then tendencia = decomp.trend, sazonal = decomp.seasonal, residual = decomp.resid. $\endgroup$
    – Murilo
    Commented Oct 27, 2022 at 12:09

2 Answers 2

1
$\begingroup$

It is often hard to disentangle the effects of autocorrelation and seasonality.

Consider the case where your time series has only day-of-week seasonality, with no autocorrelative dynamics whatsoever. Essentially, you would be drawing random observations from seven different distributions, one for each day of the week. In your ACF plot, this would show up as peaks at lags 7, 14 etc. But capturing the seasonality through seasonal dummies would be the best you could do, and adding any autocorrelation to your model would be overfitting, because the true autocorrelation parameter, after accounting for seasonality, would be zero. So it is quite conceivable that your original model, with seasonal dummies and small autocorrelation lags, may perform better than one with higher order autocorrelations.

You could investigate this by fitting your data to your weekday dummies without autocorrelations, then plotting the ACF of the residuals. If regressing on weekdays already captures most of the effect, the peaks at lag 7, 14 etc. should mostly go away.

Also, try regressing on the weekdays only. The autocorrelation may not be helpful. You do have a trend, so you could try fitting a linearly increasing trend predictor. Also, you could try a standard AutoARIMA functionality with automatic model search, you definitely have enough data for that. It's not quite clear from your plots, but you may have . There are specialized methods that can account for this. The tag wiki contains pointers to literature (and I would still try modeling the trend in addition).

Finally, you note you are using the MAPE. I assume your model does not use the MAPE in-sample as an optimization criterion, right? Also, it probably outputs expectation forecasts, not MAPE-minimal ones. Note that a MAPE-minimal forecast may be different from the expectation. However, with your large numbers, it likely does not make much of a difference (assuming you are calculating your MAPEs on the backtransformed data, not the 0-1 data - why are you scaling before fitting?). Still: What are the shortcomings of the Mean Absolute Percentage Error (MAPE)?

$\endgroup$
4
  • $\begingroup$ Thanks a lot for the valuable information! I am going to check how it works fitting only to weekday dummies without autocorrelations. Regarding MAPE, i only use it in my test set for final evaluation of performance. During training/validation i am using MSE. And yes, the MAPE is being calculated with the backtransformed data. I scaled my timeseries because after introducing the one-hot-encoded variables i ended up getting worse results than when the data was not scaled. $\endgroup$
    – Murilo
    Commented Oct 27, 2022 at 12:27
  • 1
    $\begingroup$ Training on one quality measure and evaluating on another one does not make sense, since the point forecasts that minimize MSE (which you rewarded your model to return) may be quite different from the point forecasts that minimize the MAPE. It's like throwing a stick for your dog, but punishing him for not bringing back the tennis ball. I'm rather surprised scaling your observations yielded better results - the forecasts should really be the same, whether you model & forecast directly, or transform-model-forecast-backtransform. $\endgroup$ Commented Oct 28, 2022 at 9:33
  • $\begingroup$ I understand, that's a lot i have to think and discuss with my teammates now. But should the forecasts really be the same? Scaling my time series to 0-1 made more sense, since the original data is in the scale of 10000 and introducing the one-hot-encoded variables could cause some scaling problems. $\endgroup$
    – Murilo
    Commented Oct 29, 2022 at 8:47
  • $\begingroup$ Scales of 10000 should really not cause any numerical issues. And I don't see anything else that could cause a scaling to really change things. $\endgroup$ Commented Oct 29, 2022 at 16:34
1
$\begingroup$

The first three autocorrelation coefficients in your time series are 0.7, 0.49, 0.41, which is just a little off from an AR(1) model with an autocorrelation of 0.7.

A seasonality of 7 days is also apparent from the peaks at 7, 14, 21 and 28 days.

The persistence of a significant positive autocorrelation out to 30 days and possibly beyond, is typical when a trend exists, but I do not believe this is your case, judging from the timeseries plot (top) which exhibits no trend. So it is possible that your data is better described via either integration (I1 or I2), or a long-range moving average (MA).

From the additional information in the comment, I gather that the trend plot is misleading as you fed the wrong value to the period parameter. While it is supposed to be 7 based on the autocorrelation, you fed it round(len(df) / 2), or half the length of the time series - so statsmodel was trying to find a periodicity of approximately 2.8 years in your data.

My suggestion is to correct the following in your analysis:

  1. analyze the first difference of the Load column rather than the data in the column.
  2. use period=7
  3. Turn off trend extrapolation as there should not be any trend in your data.

In code, this translates to:

df['Load_d1'] = df['Load'].diff()
decomp = seasonal_decompose(df['Load_d1'], period=7, extrapolate_trend=0)

Check the standard deviations of your residuals:

np.std(decomp.resid)

The value of your residual standard deviation should be less than the standard deviation of the original data, as the seasonality and integration components should account for most of it.

At this point, you can use linear regression on the residuals, to find the AR(1) coefficient in the de-seasonalized data. Remember that the whole thing should be attempted on Load_d1 rather than Load.

To reconstruct the prediction of Load from the differentiated timeseries, use np.cumsum().

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.