If I understood correctly, in this answer, Ben Bolker says that using inferential methods after having performed AIC-based model selection is wrong because "standard inferential methods assume the model is specified a priori". In this slide-show, Florian Hartig and Carsten Dormann furthermore suggest (last 4 slides) that there is no theory supporting the practice of inferring (computing p-values or confidence intervals (CIs)) on averaged coefficients and that reported values from R packages (like
MuMIn) are "nonsense".
In my field (ecology), a common practice is to build a set of a priori candidate models that make biological sense (i.e. with combinations of explanatory variables that we have good reasons to believe are influential on the dependent variable under study), we then select the most parsimonious ones based on a criterion (e.g. $\Delta$ AICc < 4), and we compute the unconditional averaged parameters and their 95% CIs. In R, the last steps would look like this:
Parameter <- MuMIn::model.avg(top.models, revised.var=T, adjusted=T, fit=T) Parameter.model <- as.data.frame(cbind(MuMIn::coefTable(Parameter, full=T), stats::confint(Parameter, full=T)))
As I would like to be sure to understand:
- Could someone please tell me whether computing CIs on averaged coefficients is statistically sound or not? If not, could you please explain why, as simply as possible?
- What are the alternatives if you're interested in explaining (rather than predicting) your dependent variable and you cannot afford to keep all potentially influential variables in a single model because of your limited sample size?
Thanks in advance for your helpful answers.
Before seeing Florian Hartig's enlightening answer below, I read a very interesting paper by Tredennick et al. (2021) called "A practical guide to selecting models for exploration, inference, and prediction in ecology" that further convinced me that the above practice is wrong, especially by reminding me the obvious fact that what I was trying to do is actually exploration (i.e. modelling that generates hypothesis) and not inference (i.e. modelling to test an hypothesis).
Unless I'm mistaken, their arguments highlighted two additional problems with this approach:
- i) we usually don't correct p-values for the multiple comparisons made on the same data;
- ii) by selecting models based on AIC values, we select the "best" models for prediction! These models can thus incorporate spurious relationships that help predict the outcome but do not help to understand the processes driving the outcome (which is, ultimately, the purpose of inference)! In my case, this risk is quite reduced because I wrote biologically relevant models (as opposed to testing all predictor combinations with stepwise procedures) but still, it may be problematic...
Do these criticisms make sense or am I understanding things wrong again?
To conclude, another nice part of this paper is that they admitted, at the end, that at least one of them used to do things the wrong way (mixing modelling objectives using the same dataset) so it's never too late to improve your practices... and I think that's an hopeful message.