How to refer to AIC model-averaged parameters and confidence intervals

Question

I am writing up results from regression analysis where I used AICc model averaging to arrive at my final parameter estimates. I am wondering how best to refer to these parameters and their 95% confidence intervals. It seems like "significantly different" is taboo in the AIC world, but writing out "the parameter was x.x and its CI does not cross zero" seems much more laborious to me and the reader than saying "x.x was significantly different from zero."

This seems like it might be an issue that would not come up if I had just selected the lowest AICc as my best model, which is what many folks do (against Burnham and Anderson repeatedly stating otherwise). Selecting the best model let's you say "the parameter is important b/c it is in the final model."

Also, I'm wondering if there is an AIC model averaged equivalent to "marginally significant." I have parameters that have the predicted sign, indicate a fairly sizeable effect, but whose CI creeps over 0.0.

Philosophically I like model averaging, and I also have many good models that often only differ by an extra covariate or an interaction.

EDIT: This inquiry can probably be summarized by asking "In an AICc model averaging framework how does one interpret parameters whose confidence intervals span zero by only a small amount?"

Answer 1

If you have read Burnham & Anderson's monograph, you know just why they discourage AIC(c)-based model selection: because they subscribe to the theory of tapering effect sizes. In a nutshell, they posit that everything has an effect - it's just that most effects are pretty small (sort of a "long tail"). Thus, an AIC(c)-selected model may be more parsimonious, but it will be systematically too small (the bias-variance trade-off). Therefore they recommend averaging models.

This is also the reason why statistical significance and p values are not en vogue in the Burnham & Anderson worldview. Tapering effect sizes are another way of saying that the true coefficients are almost always nonzero, just perhaps very small. Thus, the null hypothesis is already false a priori. P values pose a question that we already know the answer to.

Thus, if you follow B&A's philosophy far enough that you do AICc-based model averaging, it seems a bit incongruous to also discuss p values and/or "marginal significance".

Now, one possibility would be to simply discuss "averaged coefficients" and their CIs, without even discussing whether CIs contain zero. Conversely, if you are in a field that deifies p values (like psychology), it may make more sense to disregard these implications of B&A in the interest of talking in a way your readers will understand, rather than follow strict AICc purity.

(Anyway, my impression is that AICc and B&A have more of a following among non-statisticians, especially ecologists. So the nuances we are discussing here may already be far away from your readership's main interests.)

Answer 2

If you have access, Ive found several papers that are very helpful when deciding what to report, what values to use and the common mistakes people make when using AIC. On mistake talked about is using 95% CI when you've used AIC procedures as discussed in Arnold 2010.

Arnold T.W. 2010. Uninformative Parameters and Model Selection Using Akaike’s Information Criterion. Journal of Wildlife Management

Aurr et al 2010. A protocol for data exploration to avoid common statistical problems. Methods in Ecology and Evolution

Symonds and Moussalli 2011. A brief guide to model selection, multimodel inference and model averaging in behavioural ecology using Akaike’s information criterion

Answer 3

Use the (AICcmodavg) package in R developed by Marc J. Mazerolle. This package will allow you to compute model average estimates and their 95% confidence intervals based on your entire list of candidate models. The estimates are weighted based on the relative importance of your models (the AIC values/ranking of your models)and includes only the models for which the variable of interest is included. When computing the model average estimates, you have to be sure to exclude any interactions in which the variable of interest is included.

In your results section, include a table with each of your models and their AIC, delta AIC and AICweight values. Also include a table of the model average estimates with the variable, estimate and upper/lower 95% CI. In your write up you can then refer to the models as "The most probable model/models" and the variables "as the most important variable". If the confidence intervals of your estimates do not contain zero, then you have an effect of your variable as compared to your reference. So, If you were comparing 3 treatment types to a control and you found that only treatment 2 was different (ie. the average estimate CI did not contain 0 but treatment 1 and 3 do), instead of saying "significantly different" you could conclude that treatments 1 and 3 were similar to the control but treatment 2 had a positive effect (or negative depending on the sign of the estimate)on your response variable.

You can then use the same package to compute the predicted values for your variables using the entire list of candidate models and look at the trend. So if you were looking at the treatment effect over time, you could make predictions for treatments 1, 2, 3 and control and then say something like "Treatment 2 had a postive effect on the response variable which increased with time throughout the duration of the study period. There was no difference in response variable between treatments 1 and the control, nor treatment 3 and the control".

Answer 4

While @Stephan Kolassa's answer was probably best I'd like to just add that writing "the parameter was x.x and its CI does not cross zero" is not just laborious but treats the reader like an imbecile. When dealing with CIs simply use them as parameter estimates and if they don't cross zero that will be completely self evident when reported. A side effect is that you avoid writing that you're using your CIs solely for the purposes of inverting the t-test and therefore making absolutely no progress over such a test.