0
$\begingroup$

I try to choose the best model between the Arima model and the Feed-forward neural networks. The script runs well and I use the accuracy function to compare the to algoritm. I use the RMSEto chouse the best model, is enough or I have to compare other parametres? I'd like to precise that the data set is equal and the numbers are normalization between 0 and 1.

Below the results of the two algorithms:

ARIMA MODEL
                  ME             RMSE         MAE    MPE  MAPE    MASE        ACF1
Training set  -2.284042e-06 0.005978466 0.003094135 -Inf  Inf   1.006384  -0.0004549235
FEED FORWARD NEURAL NETWORKS MODEL
                  ME           RMSE         MAE    MPE   MAPE     MASE        ACF1
Training set -7.196957e-06 0.00571893 0.003075926 -Inf   Inf    1.000462   0.00461708

                   

In this case I chouse the Feed Forward Neural Network because the RMSE is lower respect the Arima, but my question are:

is enouth lower the RMSE?

I have to do another type of test and consider also it?

Thank in advance for any type of help.

$\endgroup$

1 Answer 1

0
$\begingroup$

One way to make sure that the difference in losses is statistically significant is to check the distributions of such metrics in the test dataset (not training as I see you are doing as long as you want the model to generalize to unseen data). You can check p-values and confidence intervals.

$\endgroup$
2
  • $\begingroup$ Thank you for your answer. $\endgroup$ Apr 16, 2020 at 10:31
  • 2
    $\begingroup$ (1) Neural networks are not estimated by maximum likelihood. (2) It is difficult to obtain an estimate of their degrees of freedom. Because of (1), LR test is not a good choice. Because of (1) and (2), AIC is not a good choice. $\endgroup$ Apr 16, 2020 at 14:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.