How to choose better model based on AICc or RMSE in certain situation?

Question

Which attribute should consider as best fitted model AICc or RMSE in auto.arima? I am having a case of

best$aicc <- Inf 
for(i in 1:25){

r = fourier(timeSeries,K=i)
ar <- auto.arima(anotherTimeseries, xreg= r, seasonal = false)
 if(ar$aicc < best$aicc){
     best = ar /* this part is reached only once for my data, the only first value is set as best and the aicc value keep on increasing*/
 }else{
    acc <- accuracy(ar)
    acc1 <- accuracy(best)
    /* check the RMSE accuracy rate and the set lowest RMSE value to best*/
 }
  forecast(best, xreg = (best i value from pervious value), h= 104)
}

Now, The doubt is whether I need to choose a best fitting model based on aicc value or RMSE value check (in the else part). Which approach will be proper?

AICC                RMSE
1642.857        acc- 233.6344
                acc1 - 234.3495

1651.623        acc- 233.3246
                acc1 - 234.3495

                acc- 232.7801
1656.273        acc1- 234.3495

RMSE value decreases in every step but AICC value increases. Which one would be better arima model ? Thanks in advance for the suggestions

Answer 1

forecast::accuracy will give you in-sample accuracy measures which are useless for model selection. By construction, a model with some additional Fourier terms will beat a model without them in sample. But we are normally interested in generalization performance, i.e. out of sample.

On the other hand, AICc estimates expected likelihood out of sample (as explained in this answer and can be found in Hastie et al. "The Elements of Statistical Learning") and is a sound criterion for model choice, especially if the goal is forecasting. You should pick the model with the lowest AICc.