Is post-variable-selection multimodel inference a bad idea?

Question

If I understood correctly, in this answer, Ben Bolker says that using inferential methods after having performed AIC-based model selection is wrong because "standard inferential methods assume the model is specified a priori". In this slide-show, Florian Hartig and Carsten Dormann furthermore suggest (last 4 slides) that there is no theory supporting the practice of inferring (computing p-values or confidence intervals (CIs)) on averaged coefficients and that reported values from R packages (like MuMIn) are "nonsense".

In my field (ecology), a common practice is to build a set of a priori candidate models that make biological sense (i.e. with combinations of explanatory variables that we have good reasons to believe are influential on the dependent variable under study), we then select the most parsimonious ones based on a criterion (e.g. $\Delta$ AICc < 4), and we compute the unconditional averaged parameters and their 95% CIs. In R, the last steps would look like this:

Parameter <- MuMIn::model.avg(top.models, revised.var=T, adjusted=T, fit=T)
Parameter.model <- as.data.frame(cbind(MuMIn::coefTable(Parameter, full=T),
                                       stats::confint(Parameter, full=T)))

As I would like to be sure to understand:

1. Could someone please tell me whether computing CIs on averaged coefficients is statistically sound or not? If not, could you please explain why, as simply as possible?
2. What are the alternatives if you're interested in explaining (rather than predicting) your dependent variable and you cannot afford to keep all potentially influential variables in a single model because of your limited sample size?

Thanks in advance for your helpful answers.

---

EDIT:
Before seeing Florian Hartig's enlightening answer below, I read a very interesting paper by Tredennick et al. (2021) called "A practical guide to selecting models for exploration, inference, and prediction in ecology" that further convinced me that the above practice is wrong, especially by reminding me the obvious fact that what I was trying to do is actually exploration (i.e. modelling that generates hypothesis) and not inference (i.e. modelling to test an hypothesis).
Unless I'm mistaken, their arguments highlighted two additional problems with this approach:

• i) we usually don't correct p-values for the multiple comparisons made on the same data;
• ii) by selecting models based on AIC values, we select the "best" models for prediction! These models can thus incorporate spurious relationships that help predict the outcome but do not help to understand the processes driving the outcome (which is, ultimately, the purpose of inference)! In my case, this risk is quite reduced because I wrote biologically relevant models (as opposed to testing all predictor combinations with stepwise procedures) but still, it may be problematic...

Do these criticisms make sense or am I understanding things wrong again?

To conclude, another nice part of this paper is that they admitted, at the end, that at least one of them used to do things the wrong way (mixing modelling objectives using the same dataset) so it's never too late to improve your practices... and I think that's an hopeful message.

Answer 1

As I am cited here, I guess it is fair that I should comment. There are two issues with calculating CIs / p-values on model-averaged coefficients.

1. The first is that model-averaged coefficients inherit all the problems that arise in calculating CIs on ensemble estimators in general. We discussed those in detail in Dormann et al., Model averaging in ecology: a review of Bayesian, information-theoretic, and tactical approaches for predictive inference https://esajournals.onlinelibrary.wiley.com/doi/10.1002/ecm.1309 (see in particular section on Quantifying uncertainty of model-averaged predictions). The bottomline of this discussion is that it is exceedingly difficult to guarantee nominal coverage on CIs of model ensembles, due to the difficulty to estimate the correlation between the estimators of the individual models, and that the existing approaches, which all make some kind of approximation, can be quite wrong.

2. The second point is specific for model-averaged coefficients, and this is what we show in the presentation: conditional and unconditional model-averaged coefficients are in general not unbiased, which further complicates the establishment of CIs with nominal coverage.