Let $X$ be a multivariate time series of $N$ variables and $T$ observations. Let split $X$ into two separate datasets :

  • $X_{train}$ : a train set with $N$ variables and $T_{train}$ observations
  • $X_{test}$ : a test set with $N$ variables and $T_{test}$ observations

We train different forecasting methods ($k$-NN, $VAR$, etc.) on $X_{train}$. Then, for each method, we perform a one step forecast on $X_{test}$.

We then want to compare the forecasting performance of the different methods. We could compute the RMSE for each variable, but given the differences in amplitude between variables: we cannot average the RMSE over the variables in order to get a unique score for each method.

I thought of two alternatives:

  1. compute MASE instead of RMSE
  2. for each forecasting methods, count the number of variables for which it has (strictly or almost) the lowest RMSE.

Do you know other ways to compare forecasting performance of several methods – either in a quantitative manner like I did or in a more qualitative manner? The number of variables is high (~600); this is why I tried to summarize the results into a single evaluation criterion.

  • $\begingroup$ What measure do you use for training error? And why is this not appropriate for test error? (For model-selection, my understanding is that most of the "extra sophistication" is essentially heuristics for comparing models of different complexity ... in the case where a test set is not available? I could be wrong!) $\endgroup$ – GeoMatt22 Nov 3 '16 at 15:08
  • $\begingroup$ The measure of training error can vary from one method to the other. For example : - for the Vector Autoregressive Model (VAR(p)), I can use an information criterion (AIC, BIC, ...) to choose the best value of meta-parameter p and the corresponding model estimation. - for $k$-NN, the parameter $k$ is usually fitted with cross-validation. The classical error measure is the residual sum of squares. But in my case, it is prone to the same problem of difference of amplitude between variables. Note : maybe I shouldn't use the tag model-selection but rather method comparison ? $\endgroup$ – Jaewon Nov 4 '16 at 9:44

It is a bit late, but try Diebold-Mariano (DM) Test. (sorry, I cannot comment because reputation is less than 50.)


  • 1
    $\begingroup$ You could improve this answer by expanding it i.e. by describing the test and why it is useful. $\endgroup$ – mkt Dec 22 '17 at 10:17

For a rolling-window forecast exercise one could also use Giacomini-White test, which is shown to be stronger than classic Diebold-Mariano test.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.