4
$\begingroup$

If I understood correctly, in this answer, Ben Bolker says that using inferential methods after having performed AIC-based model selection is wrong because "standard inferential methods assume the model is specified a priori". In this slide-show, Florian Hartig and Carsten Dormann furthermore suggest (last 4 slides) that there is no theory supporting the practice of inferring (computing p-values or confidence intervals (CIs)) on averaged coefficients and that reported values from R packages (like MuMIn) are "nonsense".

In my field (ecology), a common practice is to build a set of a priori candidate models that make biological sense (i.e. with combinations of explanatory variables that we have good reasons to believe are influential on the dependent variable under study), we then select the most parsimonious ones based on a criterion (e.g. $\Delta$ AICc < 4), and we compute the unconditional averaged parameters and their 95% CIs. In R, the last steps would look like this:

Parameter <- MuMIn::model.avg(top.models, revised.var=T, adjusted=T, fit=T)
Parameter.model <- as.data.frame(cbind(MuMIn::coefTable(Parameter, full=T),
                                       stats::confint(Parameter, full=T)))

As I would like to be sure to understand:

  1. Could someone please tell me whether computing CIs on averaged coefficients is statistically sound or not? If not, could you please explain why, as simply as possible?
  2. What are the alternatives if you're interested in explaining (rather than predicting) your dependent variable and you cannot afford to keep all potentially influential variables in a single model because of your limited sample size?

Thanks in advance for your helpful answers.


EDIT:
Before seeing Florian Hartig's enlightening answer below, I read a very interesting paper by Tredennick et al. (2021) called "A practical guide to selecting models for exploration, inference, and prediction in ecology" that further convinced me that the above practice is wrong, especially by reminding me the obvious fact that what I was trying to do is actually exploration (i.e. modelling that generates hypothesis) and not inference (i.e. modelling to test an hypothesis).
Unless I'm mistaken, their arguments highlighted two additional problems with this approach:

  • i) we usually don't correct p-values for the multiple comparisons made on the same data;
  • ii) by selecting models based on AIC values, we select the "best" models for prediction! These models can thus incorporate spurious relationships that help predict the outcome but do not help to understand the processes driving the outcome (which is, ultimately, the purpose of inference)! In my case, this risk is quite reduced because I wrote biologically relevant models (as opposed to testing all predictor combinations with stepwise procedures) but still, it may be problematic...

Do these criticisms make sense or am I understanding things wrong again?

To conclude, another nice part of this paper is that they admitted, at the end, that at least one of them used to do things the wrong way (mixing modelling objectives using the same dataset) so it's never too late to improve your practices... and I think that's an hopeful message.

$\endgroup$
5
  • $\begingroup$ I don't quite see where Ben discusses averaged coefficients. As far as I understand, he is writing about selecting a single model using, e.g., AIC, then computing confidence intervals for parameters of the selected model. Can you elaborate whether you are interested in this model selection situation, or in some model averaging method (in which case it would be good to say how the averaging of the coefficients works, and how you would compute confidence intervals on the averages)? $\endgroup$ Jan 26 at 20:31
  • $\begingroup$ Well, you are quite right, I misunderstood Ben Bolker's statement. I edited my question to better reflect what he said and what my concerns are. $\endgroup$
    – Fanfoué
    Jan 27 at 10:00
  • 3
    $\begingroup$ The OP's strategy of selecting parsimonious models is very flawed for reasons they reference. And I have reservations about doing inference on averaged coefficients. To me a cleaner, easier, and more accurate strategy that captures uncertainties is to formulate a single more general and flexible model and stick with it, whether some components of it become "insignficant" or not. Doing this in a Bayesian context with priors specifying what you know about the strength of various effects would be even better. Sometimes one has flat priors on main effects and shrinkage priors on interactions. $\endgroup$ Jan 27 at 12:56
  • $\begingroup$ @FrankHarrell, thank you. I trust your well respected opinion on this and I am more than willing to learn how to do things right. But from where I stand, three problems remain: i) I would like to understand why this approach is flawed, not least to explain to my colleagues why we should stop doing that (that's why I asked for a plain explanation); ii) with a small sample size, I cannot practice inference with a complex model, can I? And iii) for regular biologists with rather poor statistical training (like me), bayesian statistics do not seem "easy" at all... but I may be wrong on that too. $\endgroup$
    – Fanfoué
    Jan 27 at 17:52
  • 1
    $\begingroup$ Bayesian statistics is harder to learn if you've already learned traditional statistics and is easier to learn if you start from zero (witness Richard McElreath Statistical Rethinking). But it is harder to execute (although the latest computational tools are helping a lot). Bayesian methods are exact for all sample sizes, unlike traditional approaches. What is possibly wrong with what you wanted to do is that proper confidence interval coverage probability may not have been demonstrated with that approach. $\endgroup$ Jan 28 at 14:23

1 Answer 1

2
$\begingroup$

As I am cited here, I guess it is fair that I should comment. There are two issues with calculating CIs / p-values on model-averaged coefficients.

  1. The first is that model-averaged coefficients inherit all the problems that arise in calculating CIs on ensemble estimators in general. We discussed those in detail in Dormann et al., Model averaging in ecology: a review of Bayesian, information-theoretic, and tactical approaches for predictive inference https://esajournals.onlinelibrary.wiley.com/doi/10.1002/ecm.1309 (see in particular section on Quantifying uncertainty of model-averaged predictions). The bottomline of this discussion is that it is exceedingly difficult to guarantee nominal coverage on CIs of model ensembles, due to the difficulty to estimate the correlation between the estimators of the individual models, and that the existing approaches, which all make some kind of approximation, can be quite wrong.

  2. The second point is specific for model-averaged coefficients, and this is what we show in the presentation: conditional and unconditional model-averaged coefficients are in general not unbiased, which further complicates the establishment of CIs with nominal coverage.

$\endgroup$
3
  • $\begingroup$ Thank you for this helpful answer! Two remarks though: 1) I just edited my question and I would be glad to know what you think about the arguments I presented. 2) I think that in the last slide of your presentation, you inverted "unconditional MA" and "conditional MA" (the previous slides suggest that it is conditional MA that performs very poorly with collinearity and induces a strong bias, not unconditional MA). $\endgroup$
    – Fanfoué
    Feb 24 at 15:29
  • 1
    $\begingroup$ Hi, regarding your points a) I think what you're doing here is explorative inference b) for the MA estimates, I don't think the fundamental issue is multiple comparisons, this would be more for AIC selection with post-selection inference c) yes, the AIC will in general deteriorate the identification of causal effects (because collinear predictors are removed first, all other things equal). This is what I meant with bias in my point 2). $\endgroup$ Feb 25 at 12:41
  • $\begingroup$ The last issue is, however, less pronounced in the average where removed parameters are averaged with zero (full / unconditional, and yes, I think you're right, I mixed up the two in the presentation), because this average creates a shrinkage towards zero $\endgroup$ Feb 25 at 12:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.