How to compare ARIMA models with regressors?

Question

How can i compare ARIMA models that use the same forecast variable, but different regressors? I am trying to find the best regressor for the Bitcoin price. For this i have collected several time-series.

On which value should i focus? AIC, AICc and BIC or the errors ME, RMSE, MAE, ...

For example

> fit.btcPrice <- auto.arima(xts_BtcPrice[2:133], xreg=regressors[2:133,2:2])
> summary(fit.btcPrice)
Series: xts_BtcPrice[2:133] 
Regression with ARIMA(0,1,0) errors 

Coefficients:
        xreg
      0.5437
s.e.  0.1358

sigma^2 estimated as 166344:  log likelihood=-972.81
AIC=1949.61   AICc=1949.71   BIC=1955.37

Training set error measures:
                    ME     RMSE      MAE        MPE     MAPE     MASE       ACF1
Training set -25.51915 404.7512 316.8867 -0.3933149 3.646035 0.976136 -0.0592231

VS.

> fit.btcPrice <- auto.arima(xts_BtcPrice[2:133], xreg=regressors[2:133,3:4])
> summary(fit.btcPrice)
Series: xts_BtcPrice[2:133] 
Regression with ARIMA(1,0,2) errors 

Coefficients:
         ar1      ma1     ma2  intercept  twtcntLag1  sum_pos_neg
      0.9198  -0.0537  0.2823  7968.0855      0.0058       0.4679
s.e.  0.0401   0.0881  0.1082   593.5162      0.0104       0.1106

sigma^2 estimated as 162654:  log likelihood=-977.38
AIC=1968.76   AICc=1969.67   BIC=1988.94

Training set error measures:
                   ME     RMSE      MAE        MPE     MAPE      MASE        ACF1
Training set -23.7281 394.0317 308.5825 -0.4697202 3.555831 0.9505559 -0.01568309<

Answer 1

If you are more interested in in-sample fits, but still want to guard against overfitting, or in approximating the true data generating process, then you can use information criteria.

If you are more interested in forecasts, then the classical approach would be to use a holdout sample and assess out-of-sample forecast errors for your different models. If you want to check whether your different error values are statistically significantly different, the classical test is the diebold-mariano test - see its tag wiki for more information.

As to which measure of forecast accuracy (or error) to use, I recommend: Why use a certain measure of forecast error (e.g. MAD) as opposed to another (e.g. MSE)?