# Is post-variable-selection multimodel inference a bad idea?

If I understood correctly, in this answer, Ben Bolker says that using inferential methods after having performed AIC-based model selection is wrong because "standard inferential methods assume the model is specified a priori". In this slide-show, Florian Hartig and Carsten Dormann furthermore suggest (last 4 slides) that there is no theory supporting the practice of inferring (computing p-values or confidence intervals (CIs)) on averaged coefficients and that reported values from R packages (like MuMIn) are "nonsense".

In my field (ecology), a common practice is to build a set of a priori candidate models that make biological sense (i.e. with combinations of explanatory variables that we have good reasons to believe are influential on the dependent variable under study), we then select the most parsimonious ones based on a criterion (e.g. $$\Delta$$ AICc < 4), and we compute the unconditional averaged parameters and their 95% CIs. In R, the last steps would look like this:

Parameter <- MuMIn::model.avg(top.models, revised.var=T, adjusted=T, fit=T)
Parameter.model <- as.data.frame(cbind(MuMIn::coefTable(Parameter, full=T),
stats::confint(Parameter, full=T)))


As I would like to be sure to understand:

1. Could someone please tell me whether computing CIs on averaged coefficients is statistically sound or not? If not, could you please explain why, as simply as possible?
2. What are the alternatives if you're interested in explaining (rather than predicting) your dependent variable and you cannot afford to keep all potentially influential variables in a single model because of your limited sample size?

EDIT:
Before seeing Florian Hartig's enlightening answer below, I read a very interesting paper by Tredennick et al. (2021) called "A practical guide to selecting models for exploration, inference, and prediction in ecology" that further convinced me that the above practice is wrong, especially by reminding me the obvious fact that what I was trying to do is actually exploration (i.e. modelling that generates hypothesis) and not inference (i.e. modelling to test an hypothesis).
Unless I'm mistaken, their arguments highlighted two additional problems with this approach:

• i) we usually don't correct p-values for the multiple comparisons made on the same data;
• ii) by selecting models based on AIC values, we select the "best" models for prediction! These models can thus incorporate spurious relationships that help predict the outcome but do not help to understand the processes driving the outcome (which is, ultimately, the purpose of inference)! In my case, this risk is quite reduced because I wrote biologically relevant models (as opposed to testing all predictor combinations with stepwise procedures) but still, it may be problematic...

Do these criticisms make sense or am I understanding things wrong again?

To conclude, another nice part of this paper is that they admitted, at the end, that at least one of them used to do things the wrong way (mixing modelling objectives using the same dataset) so it's never too late to improve your practices... and I think that's an hopeful message.

• I don't quite see where Ben discusses averaged coefficients. As far as I understand, he is writing about selecting a single model using, e.g., AIC, then computing confidence intervals for parameters of the selected model. Can you elaborate whether you are interested in this model selection situation, or in some model averaging method (in which case it would be good to say how the averaging of the coefficients works, and how you would compute confidence intervals on the averages)? Jan 26 at 20:31
• Well, you are quite right, I misunderstood Ben Bolker's statement. I edited my question to better reflect what he said and what my concerns are. Jan 27 at 10:00
• The OP's strategy of selecting parsimonious models is very flawed for reasons they reference. And I have reservations about doing inference on averaged coefficients. To me a cleaner, easier, and more accurate strategy that captures uncertainties is to formulate a single more general and flexible model and stick with it, whether some components of it become "insignficant" or not. Doing this in a Bayesian context with priors specifying what you know about the strength of various effects would be even better. Sometimes one has flat priors on main effects and shrinkage priors on interactions. Jan 27 at 12:56
• @FrankHarrell, thank you. I trust your well respected opinion on this and I am more than willing to learn how to do things right. But from where I stand, three problems remain: i) I would like to understand why this approach is flawed, not least to explain to my colleagues why we should stop doing that (that's why I asked for a plain explanation); ii) with a small sample size, I cannot practice inference with a complex model, can I? And iii) for regular biologists with rather poor statistical training (like me), bayesian statistics do not seem "easy" at all... but I may be wrong on that too. Jan 27 at 17:52
• Bayesian statistics is harder to learn if you've already learned traditional statistics and is easier to learn if you start from zero (witness Richard McElreath Statistical Rethinking). But it is harder to execute (although the latest computational tools are helping a lot). Bayesian methods are exact for all sample sizes, unlike traditional approaches. What is possibly wrong with what you wanted to do is that proper confidence interval coverage probability may not have been demonstrated with that approach. Jan 28 at 14:23