I fitted ARIMA and NAR models to 10 different univariate time series. These time series are water characteristics at a water treatment plant (CO$_2$, TDS, TH, pH, ...).
After fitting suitable models I forecasted seven lags ahead with models for each series. I have seven observed at the end of each series that should be plotted with the corresponding seven forecasted. After plotting I found that the forecasted values by NAR model are visually near the observed but the values by ARIMA were almost constant across observed values.
For more comparison I decided to calculate MSE for the forecasted values for two models. My expectation was that MSE for NAR should be smaller than ARIMA but this was completely different. For example, for one of my series forecasted values by NAR were visually close in comparison with those from ARIMA but its MSE was more than for ARIMA.
Why? Please let me how can I compare them statistically.