How to compare multivariate forecasting methods?

Let $X$ be a multivariate time series of $N$ variables and $T$ observations. Let split $X$ into two separate datasets :

• $X_{train}$ : a train set with $N$ variables and $T_{train}$ observations
• $X_{test}$ : a test set with $N$ variables and $T_{test}$ observations

We train different forecasting methods ($k$-NN, $VAR$, etc.) on $X_{train}$. Then, for each method, we perform a one step forecast on $X_{test}$.

We then want to compare the forecasting performance of the different methods. We could compute the RMSE for each variable, but given the differences in amplitude between variables: we cannot average the RMSE over the variables in order to get a unique score for each method.

I thought of two alternatives:

1. compute MASE instead of RMSE
2. for each forecasting methods, count the number of variables for which it has (strictly or almost) the lowest RMSE.

Do you know other ways to compare forecasting performance of several methods – either in a quantitative manner like I did or in a more qualitative manner? The number of variables is high (~600); this is why I tried to summarize the results into a single evaluation criterion.

• What measure do you use for training error? And why is this not appropriate for test error? (For model-selection, my understanding is that most of the "extra sophistication" is essentially heuristics for comparing models of different complexity ... in the case where a test set is not available? I could be wrong!) – GeoMatt22 Nov 3 '16 at 15:08
• The measure of training error can vary from one method to the other. For example : - for the Vector Autoregressive Model (VAR(p)), I can use an information criterion (AIC, BIC, ...) to choose the best value of meta-parameter p and the corresponding model estimation. - for $k$-NN, the parameter $k$ is usually fitted with cross-validation. The classical error measure is the residual sum of squares. But in my case, it is prone to the same problem of difference of amplitude between variables. Note : maybe I shouldn't use the tag model-selection but rather method comparison ? – Jaewon Nov 4 '16 at 9:44