I work in the field of behavioural interventions and I use dynamic models to gain better insight into (explain) the process of behaviour change and to help inform future behavioural interventions (exploratory research using system dynamics as model-based theory building).
I currently choose state space models (SSM) as a suitable class of models I intend to explore. However, given the various flavours of SSM i.e. ARIMA, hidden Markov models, regime switching models, hybrid models; I have a couple of questions.
Should I base my modelling choice on:
- a goodness-of-fit (GoF) measure e.g. I obtain AIC and BIC for all models under consideration
- or on domain expertise e.g. based on my understanding of the domain, I expect the process to follow a stepwise (HMM) rather than a linear process (ARIMA)
Ideally, I would expect the two model selection strategies to agree. Since the choice of model (and thus, its parameters) will be used in theory building, I was wondering what would I do when they don't.
Additionally, I am aware that AIC can be used as a measure only when comparing models fitted with the same estimation method (MLE) on the same data.
Can I use information criteria such as the AIC to compare an ARIMA(p,d,q) with a HMM? (where d >0 i.e. differenced data) Or do I restrict my model space to ARMA and other flavours of SSMs. Thanks!
(I am aware of What are disadvantages of state-space models and Kalman Filter for time-series modelling?, I do not intend to comment on the features of a particular modelling framework, I am more curious to identify an objective measure that might help guide my modelling choice)