How we can select the best number of auto-regressive terms in a time series model to avoid over-fitting

Question

To find the best neural network auto regressive model for forecasting future values of a series with 180 month of observation I plotted ACF and PACF of series as follows Using Forecast package in R.

From PACF plot I decided to have 2 non- seasonal (p=2) and 3 seasonal (P=3) terms in my model using nnatar(p,P) function. Fitted values versus observed (R2=0.56) and also forecasted values for next 12 month with CI 80% and 95% are presented in following figure.

I intentionally without any logic increased the non-seasonal terms from 2 to 10 only to compare results with former model. The produced plot in this case is as follows which show the fitted values very better fitted on observed values (R2=0.99) and confidence interval for forecasted values is also pretty narrow.

My question is if selection of the number of the model predictors (seasonal and non-seasonal) should be based on PACF plot, how we can judge to select the best model while increasing the input lags will increase the model R2. I am worry about model over-fitting in second model. I am sure the behavior of my series could not be completely described by its lags. Thus certainly there may be some other exogenous variable that may have an influential effect on this series but in this model I only want to consider series lags effect. Thus I think the first model may be better selection because it described only 56% of variance but the second 99% which is not reliable! However, I am not sure about this my selection…. Please help me know what cases is better.

Answer 1

Before fitting a nnetar() model you might try the following:

1. Is the series homoscedastic? If unsure, try with option lambda="auto" to automatically select the transformation is required to stabilise the variance.

2. Fit auto.arima() (with and without lambda inside this function) to detect both seasonal and non-seasonal AR. Then use that in the nnetar() model.

3. Both nnetar() and auto.arima() has options to include exogenous series through xreg option. Might be helpful to fit models with and without this option to evaluate both point forecasts and prediction intervals.

4. Instead of fitting one single model, you might also think of fitting several models on sliding window to figure out whether one particular model is better in terms of fitting (AIC/BIC) and in terms of forecasting (MSE/MAPE etc.)

Hope this helps!