2
$\begingroup$

I have a problem understanding the parameter horizon of the function dm.test {forecast} in my particular setting.

Background:

My goal is to forecast energy consumption for individual households. The data I use reports energy load in kwh once per hour, i.e. 24 data points per day and 168 data points per week. The following example shows a typical time series over two weeks:

Data example

As the values clearly follow a seasonal pattern over the course of one day or week, my predecessors working with the same data have suggested to store one average per each of the 168 time slots in a week. Those averages can either be calculated using all available past data or with a sliding window, i.e. a moving average. For example, in order to forecast energy consumption for Monday 0:00 - 1:00 am one would average the last n observations made for that time of the week.

R Code:

I want to compare different moving averages in terms of forecast accuracy. Therefore, I calculate two vectors with different moving averages (here: ma2 and ma3) and compare them to the actual consumption values iteratively.

As there is one average per time slot, the next 168 time slots can be predicted simultaneously. Then the averages are updated using the new actual observations.

The Diebold-Mariano test included in the {forecast} package seems like an appropriate measure to test the two forecasts for statistically significant differences in prediction accuracy.

However, I am unsure whether the horizon h of my predictions in this case is 1 (as I forecast 1-step-ahead for each time slot)

dm.ma2 = dm.test((ma1-actual), (ma2-actual), h=1, power=1, alternative="two.sided")

or whether it is equal to 168 (as I make 168 predictions simultaneously)

dm.ma2 = dm.test((ma1-actual), (ma2-actual), h=168, power=1, alternative="two.sided")

I discussed the issue with two colleagues, but unfortunately we could not find a satisfactory solution. Looking at the results, h=168 seems to produce more plausible results however (for h=1 the test statistics seem inflated).

So my question is: What is the correct value for h in this context and why?

P.S.: This is my first post here. I hope I included all necessary details, but if you need additional info, feel free to comment

$\endgroup$
  • $\begingroup$ What about holiday effects ? What about detecting unusual values and neutralizing their effect ? What about trends or level shifts ? What about changes in day-of-the-week effects ?What about monthly effects ? The suggestion of the 168 model approach is in my opinion both simple and simply wrong as much more can be done turning the data into information. The problem with the 168 model approach is how to determine the best weighted average of the history of these values $\endgroup$ – IrishStat Sep 1 '15 at 14:23
1
$\begingroup$

Maybe this post can answer your question. The DM test is not suited for comparing 2 vectors containing forecasts made at one point in time for different forecasting horizons.

$\endgroup$
0
$\begingroup$

What you are doing is essentially seasonal model. Your time unit is one hour. And your seasonal frequency is 24*7=168 time units (hours). In your model there is only seasonal component (as you compare T with T-168) but there is no ordinary component (as you do not compare T with T-1).

I believe with the H parameter you are telling the function how you calculated your residuals. As I see in your code you have calculated residuals=MA1-actual then I think that your residuals are from 168 units ahead forecast. So I suggest using H=168.

$\endgroup$
0
$\begingroup$

I do not understand the whole time slot-average prediction explanation you provided, but from my knowledge of how the DM test works I can tell you that:

  • If you are making 1-step ahead forecasts (which is the same, from my POV as validating your model on a point-by-point basis), then you must use h=1
  • However, if you are doing h-step ahead forecasts with your model (that is, predicting the next h values), the DM statistic must take into account (through its modified variance estimator), that the farther a prediction is in time, the less precise it is likely to be, for there is more uncertainty about the future; and so, the 2 models' predictions are likely to differ by a greater amount (there is more dispersion on the far ends of a prediction)

For instance, if I were to predict a time series that has a trend-cycle component with a simple and Holt exponential smoothing method, the first predictions would be less uncertain on both methods (this does not imply that both methods are as precise when compared to each other, just that closer-in-time predictions are less uncertain), and become less certain and precisse as the series evolves:

1-step ahead forecasts method comparison

Here is an image of 2-step ahead forecasts for 4 different prediction methods

The resulting two-sided p-value of a DM test on the first and fourth predictions are around 0.20, even though one can clearly see that the 4th method performs better on the whole forecast horizon than for instance the first.

What happenes is, that the DM test as h increases is taking into account that the farther in time the prediction value is, the more imprecise it will be (just like the confidence intervals in a prediction); and so, the the two methods yield simmilar results for small h, but when we increase it the test makes those differences "less important" because the are naturally less precise.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.