I try to choose the best model between the Arima model and the Feed-forward neural networks. The script runs well and I use the accuracy function to compare the to algoritm. I use the RMSEto chouse the best model, is enough or I have to compare other parametres? I'd like to precise that the data set is equal and the numbers are normalization between 0 and 1.

Below the results of the two algorithms:

                  ME             RMSE         MAE    MPE  MAPE    MASE        ACF1
Training set  -2.284042e-06 0.005978466 0.003094135 -Inf  Inf   1.006384  -0.0004549235
                  ME           RMSE         MAE    MPE   MAPE     MASE        ACF1
Training set -7.196957e-06 0.00571893 0.003075926 -Inf   Inf    1.000462   0.00461708


In this case I chouse the Feed Forward Neural Network because the RMSE is lower respect the Arima, but my question are:

is enouth lower the RMSE?

I have to do another type of test and consider also it?

Thank in advance for any type of help.


1 Answer 1


One way to make sure that the difference in losses is statistically significant is to check the distributions of such metrics in the test dataset (not training as I see you are doing as long as you want the model to generalize to unseen data). You can check p-values and confidence intervals.

  • $\begingroup$ Thank you for your answer. $\endgroup$ Commented Apr 16, 2020 at 10:31
  • 2
    $\begingroup$ (1) Neural networks are not estimated by maximum likelihood. (2) It is difficult to obtain an estimate of their degrees of freedom. Because of (1), LR test is not a good choice. Because of (1) and (2), AIC is not a good choice. $\endgroup$ Commented Apr 16, 2020 at 14:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.