ARMA model selection: in-sample vs. out-sample accuracy measures I have a time series for 1000 days for many firms. I am interested to know, in general, on what basis I should select an ARMA model (the nature of my problem restricts integration order to 0). Should it be based on a predictive measure (e.g. MAE, RMSE) or AIC/BIC (measures of the closeness of a model to the unknown data generation process)?
 A: There is no very strong agreement among forecasters or time series modelers on how to choose a model.


*

*If you are more interested in forecasting, I would recommend using a holdout sample: hold back the last, say, 100 data points, fit your models to the first 900 data points, forecast out into your holdout period and examine the quality of this forecast using whatever accuracy measure makes sense in your application. Pick the model that performs best in the holdout sample.

*If you are more interested in understanding the historical dynamics (maybe you are only interested in detecting change points or such), you could also use information criteria like AIC or BIC.
That said, often forecasters use information criteria, or time series modelers may use holdout samples, arguing that you can only claim you have understood a time series well if you can forecast it well.


*

*Or you could do the Box-Jenkins procedure and look at correlograms, ACF or PACF. This question may be helpful.
You may profit from reading through previous questions tagged both "ARMA" and "model-selection", or both "ARIMA" and "model-selection". 
