Accuracy : Arima Vs Feed-forward neural networks

Question

I try to choose the best model between the Arima model and the Feed-forward neural networks. The script runs well and I use the accuracy function to compare the to algoritm. I use the RMSEto chouse the best model, is enough or I have to compare other parametres? I'd like to precise that the data set is equal and the numbers are normalization between 0 and 1.

Below the results of the two algorithms:

ARIMA MODEL

                  ME             RMSE         MAE    MPE  MAPE    MASE        ACF1
Training set  -2.284042e-06 0.005978466 0.003094135 -Inf  Inf   1.006384  -0.0004549235

FEED FORWARD NEURAL NETWORKS MODEL

                  ME           RMSE         MAE    MPE   MAPE     MASE        ACF1
Training set -7.196957e-06 0.00571893 0.003075926 -Inf   Inf    1.000462   0.00461708

                   

In this case I chouse the Feed Forward Neural Network because the RMSE is lower respect the Arima, but my question are:

is enouth lower the RMSE?

I have to do another type of test and consider also it?

Thank in advance for any type of help.

Answer 1

One way to make sure that the difference in losses is statistically significant is to check the distributions of such metrics in the test dataset (not training as I see you are doing as long as you want the model to generalize to unseen data). You can check p-values and confidence intervals.