Skip to main content

Questions tagged [diebold-mariano-test]

A test for superior predictive accuracy of one forecast over another.

Filter by
Sorted by
Tagged with
2 votes
1 answer
52 views

Diebold-Mariano test for one-step forecasts using Mean Absolute Scaled Error

This is my first time doing time series forecasting, so I am sorry for any inconsistencies in my question. But I have two different models that I want to compare. On Wikipedia, I read about Mean ...
Oskar Weber's user avatar
1 vote
1 answer
65 views

Comparing forecasts under overlapping rolling evaluation windows

Two models with differing assumptions each purport to provide a forecast for a vector of values 1 year in the future. Each model has been run at the start of each month for three years: Run Date ...
David's user avatar
  • 515
0 votes
1 answer
95 views

Statistically testing MSE difference between models predictions without normality of squared residuals

I have a dataset that I have splitted for training - testing purpose (~2400 - ~600) . After training an XGBoost regression model on the training set, I would like to statistically test the fact that ...
Guilhem Nespoulous's user avatar
2 votes
1 answer
130 views

Diebold-Mariano test for evaluating 40 different forecast horizons

I have a question about the Diebold-Mariano test. I have different forecasting horizons (n-ahead = 1 to 40) and different forecasting origins (26). I.e. I employ a rolling origin evaluation approach. ...
An economist's user avatar
0 votes
1 answer
305 views

How to interpret modified Diebold and Mariano test?

I would appreciate if you could let me know to conclude about the modified Diebold and Mariano test when our alternative hypothesis is less? How about greater? ...
ebrahimi's user avatar
  • 291
1 vote
1 answer
45 views

Compare the predictive capacity of multivariate models

I want to compare the prediction capacity of multivariate models (MGARCH, VAR, SVAR, VECM...). One option to do this is by comparing its error metrics: the mean square error or any other between the ...
Javier Andres Marino Villalba's user avatar
1 vote
1 answer
78 views

Is ARIMA-GARCH nested within ARIMA?

I wanted to compare ARIMA(1,1,1)-GARCH(1,1) and ARIMA(1,1,1) model forecasts with a Diebold-Mariano test, but I know that it cannot be used for nested models. Is ARIMA-GARCH technically nested within ...
Ula 's user avatar
  • 25
0 votes
0 answers
41 views

Statistical test for forecast performance over multiple runs

Lets say I have a time series, create a training and test set, and I want to compare the predictive accuracy of two models, by measuring e.g. the mean absolute error (MAE) over the test set. I know ...
Akylas Stratigakos's user avatar
1 vote
0 answers
56 views

How to compare different models in their ability to forecast the value-at-risk with Diebold-Mariano test?

I made value-at-risk forecasts for different models for the 95, 97.5 and 99%. I also made a dummy which equals 1 if the true return was below the value-at-risk, 0 otherwise. How can I compare those ...
Elise's user avatar
  • 51
0 votes
0 answers
156 views

Diebold-Mariano test for comparing multivariate models

I want to know if there exists a Diebold-Mariano test version for comparing the accuracy of predictions between multivariate statistics models. For example, compare a VAR model and multivariate GARCH ...
user.'s user avatar
  • 211
1 vote
1 answer
809 views

Meaning of Diebold-Mariano (DM) test for other accuracy measures (MDA, $R^2$...)

I am trying to compare the accuracies of two time series forecasts. I read about the Diebold-Mariano (DM) Test, which tests the null hypothesis of $E[d_t]=0$, where $d_t=g(e_{it})-g(e_{jt})$ is the ...
M. Sullivan's user avatar
0 votes
1 answer
31 views

Vector valued time series forecast significance testing?

I have two vector valued time series forecasts, and the components of the vector can be correlated. How do i significance test (ex. diebold mariano) for which is the better forecaster? I found two ...
Dhamma K's user avatar
1 vote
1 answer
765 views

time series Diebold-Mariano Test n-ahead forecast in R

I have a question about "h" in dm.test & DM.test from package forecast & multDM, assume I set a ARIMA model to forecast n-ahead = 20 (using dynamic regression not one step forecast), ...
Ivan's user avatar
  • 103
1 vote
0 answers
76 views

CLT in Diebold & Mariano (1995)

The Diebold-Mariano (DM) statistic is derived as follows: Assuming the loss-differential between the two models $d_t$ is covariance stationary: $$\begin{cases} \mathbb{E}[d_t] = \mu> 0 & \...
Grada Gukovic's user avatar
9 votes
2 answers
1k views

How can I determine that a forecast is significantly more accurate than another one? (time series)

Reproducible Example Look at this reproducible example: I have a time series that I want to forecast. For the sake of reproducibility, I'll just take AirPassanger. ...
Edo's user avatar
  • 193
3 votes
1 answer
343 views

Diebold-Mariano test for non time-series data

Can the Diebold Mariano test be used to compare the MSE of two models (trained on the same data) for non time series data?
Evieee's user avatar
  • 31
4 votes
2 answers
756 views

How to test superior predictive ability over multiple time series?

Suppose you have two models, model A and model B, and let these models forecast 10 time series over a horizon of 12 periods. That is, suppose the time series contain monthly data and your forecasting ...
Whizkid95's user avatar
  • 292
0 votes
1 answer
379 views

Difference between h and T in Harvey-adjusted Diebold and Mariano test?

I am looking into comparing the predictive accuracy of forecasts of different models against a benchmark model. For this I have looked into the Diebold-Mariano test statistic. However, I am using the ...
kristoffer3110's user avatar
1 vote
0 answers
487 views

Diebold-Mariano test for predictive accuracy with R

The following example code is from the documentation of the command dm.test from R. ...
user63841219's user avatar
1 vote
1 answer
303 views

Comparing multiple models with something like a Diebold-Mariano test

I run many models on different samples (i.e., datasets which are slightly different from each other). I calculate the MSE for each model on these datasets. Now I want to compare these models to see if ...
Manal's user avatar
  • 11
0 votes
0 answers
42 views

Statistical test for forecasting models

I developed many forecasting models over the same dataset (multiple iteration of simulated time series data). My dataset basically is a multivariate timeseries so the forecasting models forecast many ...
Manal's user avatar
  • 11
1 vote
1 answer
604 views

Should I use the Diebold-Mariano test year-by-year or on the overall forecast?

I have built two models, one ARIMAX and one VAR, to compare against a baseline ARIMA model to predict a weekly economic time series of interest. I am primarily comparing the accuracy of my models ...
laborEcon's user avatar
4 votes
1 answer
830 views

t-test for time series (Diebold Mariano test?)

Currently I am working on the following problem: I want to compare the means of two different time series ts1 and ts2. As the samples are not iid and t-test can't be applied, I thought to use the ...
Simon's user avatar
  • 41
5 votes
1 answer
1k views

Alternatives for Diebold-Mariano test when comparing the best forecast among many against a benchmark

Suppose I encounter a new forecasting method and I wish to evaluate it against a benchmark. I can obtain forecasts from the two methods and compare them to actual realizations and thus obtain the ...
Richard Hardy's user avatar
1 vote
0 answers
174 views

Is it possible to bootstrap a Diebold-Mariano test?

I am currently working on a small project where I want to use a (two-sided) 1-step ahead ($h=1$) Diebold Mariano test to compare forecast losses for different realized measures calculated on time-...
user262734's user avatar
0 votes
0 answers
232 views

Diebold Mariano test Nested Models

I have computed forecasts with 4 different methods, namely OLS, Elastic Net, Cubic splines in combination with Lasso, and Neural Network. All models use the same set of base variables, except cubic ...
John's user avatar
  • 277
2 votes
1 answer
369 views

Can I give continuous rank probability score (CRPS) to Diebold-Mariano (DM) test?

I would like to use DM test for probabilistic forecasting case. My initial thinking was to give CRPS of two forecasting methods instead of raw forecast errors, where CRPS is calculated using ...
Reiso's user avatar
  • 68
2 votes
1 answer
85 views

Combining the conclusions from multiple Diebold-Mariano tests

Suppose I have two forecasting models, A and B. Moreover, suppose I have multiple datasets and I use each of these sets to perform Diebold-Mariano tests. My aim is to find out which of the two models ...
user120911's user avatar
0 votes
0 answers
465 views

Autocorrelation in loss function, want to perform DM test. How many lags to use? (R)

I have two sets of forecasting errors, and want to perform a DM test. Both forecasts are a fixed size moving window, and are 1 day ahead forecasts. The first step of performing the DM test is to ...
TheManR's user avatar
  • 93
7 votes
1 answer
4k views

How to test if two RMSE are significantly different?

Say I have two models for a regression task and from each model I get a RMSE. One RMSE is smaller than the other, however I wish to test if the difference is statistically significant in order to be ...
Lay González's user avatar
2 votes
1 answer
513 views

Diebold-Mariano Test (Newey–West adjusted)

I've two questions regarding to the Diebold-Mariano test in comparing predictive acuraccy. I am reading this paper here. Differences in MSPE are reported together with p-values from the Diebold– ...
rook1996's user avatar
  • 485
2 votes
1 answer
314 views

Can i use Diebold Mariano test for comparison of 2 models across multiple time series?

I have 2 models (for simplicity, let's call them AR(1) and MA(1)) each making 1 day ahead forecasts of time series. If I had only 1 time series I would just use ...
ira's user avatar
  • 451
3 votes
1 answer
424 views

Diebold-Mariano for in-sample model comparisons?

Suppose I have two in-sample forecasts from two different non-nested models. I want to check which one produces the best forecasts. A common way is Diebold-Mariano, GiacominiWhite, ENC-T test. However,...
Daniel Pinto's user avatar
0 votes
1 answer
235 views

Comparing Forecast Errors

I want to compare my VAR-forecast to an ARMA-benchmark. The VAR forecast has a smaller MPSE (mean squared prediction error) and I want to check for significance. For this purpose I applied the Diebold-...
Niels's user avatar
  • 1
1 vote
1 answer
206 views

Diebold-Mariano with multiple predictions over time

I've been trying to use a DM test on my predictive model. I have two different models. Each model is calibrated by using a MSE loss function to estimate the price of 374 different assets each day. I ...
JohanWikstrom's user avatar
3 votes
1 answer
4k views

Diebold-Mariano in the context of volatility forecasting: What is the ultimate aim of this test?

Perhaps I'm missing a simple conceptual point here. But do the error statistics (RMSE, MAE) not tell which is the best forecast by presenting the lowest figures between the forecast and the actual ...
Albe's user avatar
  • 119
2 votes
1 answer
919 views

Tests of Forecast Accuracy for Nested Models

Can anyone explain why "classic" tests of forecast accuracy, (i.e. Diebold-Mariano test, Meese-Rogoff test and Morgan-Granger-Newbold test) are not suited for nested models? I could not find a good ...
mlieb's user avatar
  • 21
1 vote
1 answer
2k views

Compute the p-value of Diebold-Mariano Test

I am using Diebold-Mariano Test for testing the equal predictive accuracy of two models. I use the code written by Semin Ibisevic (2011) in MATLAB to compute it ...
esem's user avatar
  • 13
1 vote
0 answers
394 views

Diebold-Mariano test in Stata, autocovariance denominator doubt

I'm trying to reproduce the Diebold&Mariano test available in Stata13 and I think there is an error in the suggested implementation. At the bottom of the file you have: ...
user143645's user avatar
4 votes
1 answer
2k views

Diebold-Mariano with RMSE as loss function?

When applying the Diebold-Mariano test to test for predictive accuracy we need to specify a loss differential. For instance the loss differential $d$ in terms of the mean absolute error (MAE) is $d = ...
Alexander De Beur's user avatar
1 vote
1 answer
3k views

Diebold-Mariano, but how to say which one is more accurate?

I know that the Diebold-Mariano test is simply testing whether two forecasts are likely to forecast with the same accuracy (null hypothesis) or not (reject the null) based on some loss function. But ...
Alexander De Beur's user avatar
1 vote
0 answers
40 views

Test if one forecast outperforms all other

I have 4 different models to forecast some quantity of interest y With the Diebold-Mariano Test I can test if one model is predicting more accurate than another, which is cool. So I can make some 4x4 ...
Plazi's user avatar
  • 277
2 votes
1 answer
3k views

Diebold-Mariano test for multiple prediction horizons

I am trying to compare two forecasts using the Mariano Diebold test in R. Both forecasts are for 150 days ahead; that is, on day $t$ I forecast $t+1, t+2, \dotsc, t+150$. I deduced from this post ...
Marieke's user avatar
  • 33
2 votes
1 answer
3k views

Diebold-Mariano test in case of nested models (Clark & McCracken, 2001)

I have become aware of Clark & McCracken (2001) showing that the asymptotics of the Diebold-Mariano test will potentially collapse when comparing forecast accuracy of nested models (such as GARCH /...
Pedestrian's user avatar
0 votes
0 answers
870 views

Diebold-Mariano Test in R - how to rationalize p-value?

Consider 3 models (model1, model2, model3) with the following set of prediction errors: ...
kanimbla's user avatar
  • 584
2 votes
1 answer
1k views

Diebold-Mariano test: robust standard errors for 1-day ahead forecasts?

Diebold & Mariano suggest using Newey-West standard errors to correct for autocorrelation and heteroscedasticity in the error terms when comparing forecast accuracy. However, if I understand ...
Pedestrian's user avatar
1 vote
1 answer
6k views

Diebold Mariano test (in R)

As I asked in my answer to this question: does anyone know if the DM test (in R in this case) is supposed to be made with h=h-1? If not, am I supposed to make several prediction sets (with h ...
Miguel M.'s user avatar
  • 548
0 votes
1 answer
679 views

Diebold - Mariano test for volatility forecasts problem

I am using packages {rugarch} for forecasting and {forecast} for Diebold - Mariano test. As a first step, I am specifying the first AR-GARCH model for financial time series (AAPL Nasdaq) using ...
John's user avatar
  • 1
4 votes
1 answer
487 views

Difference between comparing forecasts and models

I started out looking for a way to test the difference between MSPE between two models (Question here), when (thanks to @Richard Hardy) I ended up reading a paper of Diebold regarding the Diebold-...
Erosennin's user avatar
  • 1,734
5 votes
1 answer
2k views

ANOVA / t-test to compare the errors of different models

I have two forecasting models, moving average and single exponential smoothing. The values of Mean Absolute Percentage Error (MAPE) is 5.2%, 5.8%. Since the difference of MAPE between the models are ...
Mukund's user avatar
  • 225