Assuming I have a series of nested non-linear models applied to a multiple populations: what approaches can I use for comparing results of models with variations in the number of predictors? The models I am using are series of variations in a basic model that has a term for linear time and seasonality. Other models include new terms added to basic model including nonlinear time and interactions plus intervention terms (dummy variable) and other continuous predictors. I BELIEVE this is referred to as a series of nested models. In all, there would be a maximum of 8 models per population.

If so, what test criteria can I use to compare within and between populations assuming of course I am modeling the same response variables for a given population and have the same n size for each model. What is the best way to make model selections for a given population and if possible to compare results across multiple populations for the same response variable?

What I am looking for what I would call would be a "consensus" or "average" model generally applicable across the multiple populations that I am analyzing. I do not necessarily want the model that is the "best predictive model" but rather the most parsimonious model that provides me with best explanation of the data i.e. what is causing the underlying pattern in the data. In other words, I am not interested in getting the "best fit" by minimizing AIC if I am only explaining a small percentage of the variance. Best predictive model may do that or it may not do that. AIC can go down simply because you add variables same as R-squared increasing. Is there an approach for determining whether or not a particular model is universally applicable across multiple populations? What about comparing the frequencies of models selected against a predicted frequency given an equal probability of any single model being the correct one using something like a Z-test?

And here is another question, what should be done when AIC decreases and predictors are not significant? The information-theoretic approach papers I've read suggest not to use p-values in combination with AIC and simply use Akaike weights. But I've read elsewhere that patterns in Delta AIC and p-value are VERY closely related (see Murtaugh, 2014 Ecology, 95(3), 2014, pp. 611–617).

I know this question has been asked on this forum before and answered to varying degrees; however and with all due respect to previous posts, not to my s atisfaction. That is why I am asking it again from the perspective of someone NOT necessarily familiar with all of the background mathematics and literature. Any answers would be appreciated.

  • $\begingroup$ AIC tends to overfit, while Bayesian Information Criterion (BIC) tends to underfit. So the truth shall lie somewhere in between. You may apply both criterions and interpret their outcomes for the case at hand. $\endgroup$ – corey979 Mar 26 at 20:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.