To find the best neural network auto regressive model for forecasting future values of a series with 180 month of observation I plotted ACF and PACF of series as follows Using Forecast package in R.
From PACF plot I decided to have 2 non- seasonal (p=2) and 3 seasonal (P=3) terms in my model using nnatar(p,P) function. Fitted values versus observed (R2=0.56) and also forecasted values for next 12 month with CI 80% and 95% are presented in following figure.
I intentionally without any logic increased the non-seasonal terms from 2 to 10 only to compare results with former model. The produced plot in this case is as follows which show the fitted values very better fitted on observed values (R2=0.99) and confidence interval for forecasted values is also pretty narrow.
My question is if selection of the number of the model predictors (seasonal and non-seasonal) should be based on PACF plot, how we can judge to select the best model while increasing the input lags will increase the model R2. I am worry about model over-fitting in second model. I am sure the behavior of my series could not be completely described by its lags. Thus certainly there may be some other exogenous variable that may have an influential effect on this series but in this model I only want to consider series lags effect. Thus I think the first model may be better selection because it described only 56% of variance but the second 99% which is not reliable! However, I am not sure about this my selection…. Please help me know what cases is better.