Questions tagged [diebold-mariano]

A test for superior predictive accuracy of one forecast over another.

Filter by
Sorted by
Tagged with
0
votes
0answers
17 views

Comparing two tests: Diebold-Mariano vs. Giacomini-White

What is (are) the main difference(s) between the Diebold-Mariano and the Giacomini-White tests of superior predictive ability? When does (do) the difference(s) matter?
1
vote
1answer
42 views

Should I use the Diebold-Mariano test year-by-year or on the overall forecast?

I have built two models, one ARIMAX and one VAR, to compare against a baseline ARIMA model to predict a weekly economic time series of interest. I am primarily comparing the accuracy of my models ...
2
votes
1answer
51 views

t-test for time series (Diebold Mariano test?)

Currently I am working on the following problem: I want to compare the means of two different time series ts1 and ts2. As the samples are not iid and t-test can't be applied, I thought to use the ...
2
votes
1answer
67 views

Alternatives for Diebold-Mariano test when comparing the best forecast among many against a benchmark

Suppose I encounter a new forecasting method and I wish to evaluate it against a benchmark. I can obtain forecasts from the two methods and compare them to actual realizations and thus obtain the ...
1
vote
0answers
41 views

Is it possible to bootstrap a Diebold-Mariano test?

I am currently working on a small project where I want to use a (two-sided) 1-step ahead ($h=1$) Diebold Mariano test to compare forecast losses for different realized measures calculated on time-...
0
votes
0answers
44 views

Diebold Mariano test Nested Models

I have computed forecasts with 4 different methods, namely OLS, Elastic Net, Cubic splines in combination with Lasso, and Neural Network. All models use the same set of base variables, except cubic ...
2
votes
1answer
107 views

Can I give continuous rank probability score (CRPS) to Diebold-Mariano (DM) test?

I would like to use DM test for probabilistic forecasting case. My initial thinking was to give CRPS of two forecasting methods instead of raw forecast errors, where CRPS is calculated using ...
0
votes
0answers
92 views

Autocorrelation in loss function, want to perform DM test. How many lags to use? (R)

I have two sets of forecasting errors, and want to perform a DM test. Both forecasts are a fixed size moving window, and are 1 day ahead forecasts. The first step of performing the DM test is to ...
2
votes
1answer
879 views

How to test if two RMSE are significantly different?

Say I have two models for a regression task and from each model I get a RMSE. One RMSE is smaller than the other, however I wish to test if the difference is statistically significant in order to be ...
1
vote
0answers
119 views

Can i use Diebold Mariano test for comparison of 2 models across multiple time series?

I have 2 models (for simplicity, let's call them AR(1) and MA(1)) each making 1 day ahead forecasts of time series. If I had only 1 time series I would just use ...
1
vote
1answer
181 views

Diebold-Mariano for in-sample model comparisons?

Suppose I have two in-sample forecasts from two different non-nested models. I want to check which one produces the best forecasts. A common way is Diebold-Mariano, GiacominiWhite, ENC-T test. However,...
0
votes
1answer
161 views

Comparing Forecast Errors

I want to compare my VAR-forecast to an ARMA-benchmark. The VAR forecast has a smaller MPSE (mean squared prediction error) and I want to check for significance. For this purpose I applied the Diebold-...
1
vote
1answer
84 views

Diebold-Mariano with multiple predictions over time

I've been trying to use a DM test on my predictive model. I have two different models. Each model is calibrated by using a MSE loss function to estimate the price of 374 different assets each day. I ...
3
votes
1answer
477 views

Diebold-Mariano in the context of volatility forecasting: What is the ultimate aim of this test?

Perhaps I'm missing a simple conceptual point here. But do the error statistics (RMSE, MAE) not tell which is the best forecast by presenting the lowest figures between the forecast and the actual ...
2
votes
1answer
402 views

Tests of Forecast Accuracy for Nested Models

Can anyone explain why "classic" tests of forecast accuracy, (i.e. Diebold-Mariano test, Meese-Rogoff test and Morgan-Granger-Newbold test) are not suited for nested models? I could not find a good ...
1
vote
1answer
880 views

Compute the p-value of Diebold-Mariano Test

I am using Diebold-Mariano Test for testing the equal predictive accuracy of two models. I use the code written by Semin Ibisevic (2011) in MATLAB to compute it ...
1
vote
0answers
211 views

Diebold-Mariano test in Stata, autocovariance denominator doubt

I'm trying to reproduce the Diebold&Mariano test available in Stata13 and I think there is an error in the suggested implementation. At the bottom of the file you have: ...
4
votes
1answer
599 views

Diebold-Mariano with RMSE as loss function?

When applying the Diebold-Mariano test to test for predictive accuracy we need to specify a loss differential. For instance the loss differential $d$ in terms of the mean absolute error (MAE) is $d = ...
0
votes
1answer
1k views

Diebold-Mariano, but how to say which one is more accurate?

I know that the Diebold-Mariano test is simply testing whether two forecasts are likely to forecast with the same accuracy (null hypothesis) or not (reject the null) based on some loss function. But ...
1
vote
0answers
34 views

Test if one forecast outperforms all other

I have 4 different models to forecast some quantity of interest y With the Diebold-Mariano Test I can test if one model is predicting more accurate than another, which is cool. So I can make some 4x4 ...
2
votes
1answer
1k views

Diebold-Mariano test for multiple prediction horizons

I am trying to compare two forecasts using the Mariano Diebold test in R. Both forecasts are for 150 days ahead; that is, on day $t$ I forecast $t+1, t+2, \dotsc, t+150$. I deduced from this post ...
0
votes
1answer
1k views

Diebold-Mariano test in case of nested models (Clark & McCracken, 2001)

I have become aware of Clark & McCracken (2001) showing that the asymptotics of the Diebold-Mariano test will potentially collapse when comparing forecast accuracy of nested models (such as GARCH /...
0
votes
0answers
313 views

Diebold-Mariano Test in R - how to rationalize p-value?

Consider 3 models (model1, model2, model3) with the following set of prediction errors: ...
2
votes
1answer
672 views

Diebold-Mariano test: robust standard errors for 1-day ahead forecasts?

Diebold & Mariano suggest using Newey-West standard errors to correct for autocorrelation and heteroscedasticity in the error terms when comparing forecast accuracy. However, if I understand ...
0
votes
1answer
3k views

Diebold Mariano test (in R)

As I asked in my answer to this question: does anyone know if the DM test (in R in this case) is supposed to be made with h=h-1? If not, am I supposed to make several prediction sets (with h ...
0
votes
1answer
332 views

Diebold - Mariano test for volatility forecasts problem

I am using packages {rugarch} for forecasting and {forecast} for Diebold - Mariano test. As a first step, I am specifying the first AR-GARCH model for financial time series (AAPL Nasdaq) using ...
3
votes
1answer
267 views

Difference between comparing forecasts and models

I started out looking for a way to test the difference between MSPE between two models (Question here), when (thanks to @Richard Hardy) I ended up reading a paper of Diebold regarding the Diebold-...
3
votes
1answer
663 views

ANOVA / t-test to compare the errors of different models

I have two forecasting models, moving average and single exponential smoothing. The values of Mean Absolute Percentage Error (MAPE) is 5.2%, 5.8%. Since the difference of MAPE between the models are ...
1
vote
1answer
576 views

On a problem with the implementation of the test of Diebold and Mariano for equal predictive accuracy

To recall how the test of Diebold and Mariano is set up I utilize the parsimonious words of Diebold himself: Now, I have been trying to perform this test on the respective vector of errors of two ...
2
votes
1answer
154 views

Using Diebold-Mariano test to compare predictive errors in non-time-series?

I understand that the DM test is established for time series data, but could I still apply the test for non-time-series data? Could I simply replace the autocorvaiance part of the test statistics with ...
7
votes
1answer
7k views

Diebold-Mariano test for predictive accuracy

I am using the Diebold-Mariano test in the forecast package of R to test the predictive accuracy. In particular, I want to ...
2
votes
3answers
2k views

Understanding forecast horizon for Diebold-Mariano tests

I have a problem understanding the parameter horizon of the function dm.test {forecast} in my particular setting. Background: My goal is to forecast energy consumption for individual households. The ...