Which factor should I consider to choose the best model, stationarity or lowest MSE?

Question

Recently, I did a project where I modelled a dataset of Gold price. I used ARIMA(1,1,1) and ARIMA(2,2,3) to model the data. The results that I get was ARIMA(1,1,1) was not stationary but the MSE value is low (118) and the ARIMA(2,2,3) is stationary but the MSE is high (170).

Answer 1

ARIMA(1,1,1) is a nonstationary process as it contains a (single) unit root; the order of integration (the middle number in the parentheses) tells you that. ARIMA(2,2,3) also is a nonstationary process as it contains two unit roots. However, in neither case is this a fundamental problem; there are well-known estimators of the model's parameters that have favorable properties (consistency, asymptotic normality) that are implemented in numerous statistical packages. What should concern you, on the other hand, is the stationarity of the error term. You should examine the models' residuals (proxies for the errors) and see that they are i.i.d. as a precondition for trusting your models.

If you have chosen to judge your models by MSE, you must be facing square loss. If that is the case, lower out-of-sample MSE is better, and a model delivering that can be preferred.* If you are not facing square loss (but e.g. absolute loss), consider using an appropriate loss function / error metric (e.g. mean absolute error) for judging your models' out-of-sample performance and for comparisons across models.

*In-sample MSE can be a poor measure of prediction quality, especially for models with many parameters. It is also a poor measure for model comparison, as it favours models with many parameters regardless of their actual out-of-sample forecasting ability. Information criteria can remedy that; comparing models' AICs or BICs therefore can make more sense than comparing in-sample MSEs.