You should always be checking the statistical significance of any estimated parameter and evaluating the error process for any remaining structure ( or structure that was injected due to a bad arima model ). This structure could be further arima , pulses , level shifts , seasonal pulses ,local time trends , changes in error variance over time possibly a symptom of changing parameters . deterministic error variance change at one or more points or the need for a Box-Cox transformation. Simple methods (aic/bic) assuming a list of possible models premise that all of the possible violations are not present. The possible violations should always be tested for by tests on the error process. My first attempt to automate model identification in 1968 was to try some 30 or so models to try and to "pick the best" . That procedure required modification !

Trust but Verify !You should always be checking the statistical significance of any estimated parameter and evaluating the error process for any remaining structure ( or structure that was injected due to a bad arima model ). This structure could be further arima , pulses , level shifts , seasonal pulses ,local time trends , changes in error variance over time possibly a symptom of changing parameters . deterministic error variance change at one or more points or the need for a Box-Cox transformation. Simple methods (aic/bic) assuming a list of possible models premise that all of the possible violations are not present. The possible violations should always be tested for by tests on the error process. My first attempt to automate model identification in 1968 was to try some 30 or so models to try and to "pick the best" . That procedure required modification !