Model averaging: predictor significance vs importance

Question

What is the correct way to report the results of model averaging? I’m using MuMin in R, results include a p-value. It’s also possible to get the importance of each predictor. Example results below.

What exactly is the difference and how do I interpret this? What does it mean, for example, if a predictor is important but not significant, or vice versa? And which is best to report?

Example results below from dredge() with a mixed effects model fit in lme4:

Model-averaged coefficients:
(full average) 
                          Estimate Std. Error Adjusted SE z value Pr(>|z|)    
(Intercept)                0.56610    0.24118     0.24233   2.336 0.019491 *  
Var1                      -0.91115    0.23856     0.23968   3.802 0.000144 ***
Var5                       0.08168    0.23656     0.23769   0.344 0.731132    
Var7                       0.22617    0.31847     0.31918   0.709 0.478575    
Var8                       0.11879    0.18559     0.18635   0.637 0.523808    
Var3                      -1.02662    0.37856     0.38034   2.699 0.006950 ** 
Var2                      -0.14402    0.23801     0.23868   0.603 0.546239    
Var4                      -1.03832    0.34015     0.34174   3.038 0.002379 ** 
Var1:Var8                  0.37484    0.28242     0.28289   1.325 0.185163    
Var1:Var3                 -0.92612    0.49060     0.49291   1.879 0.060262 .  
Var5:Var4                  0.65352    0.25435     0.25557   2.557 0.010556 *  
Var3:Var4                 -1.48070    0.66730     0.67048   2.208 0.027215 *  
Var9                      -0.05323    0.14171     0.14212   0.375 0.707990    
Var1:Var9                 -0.13693    0.24729     0.24749   0.553 0.580071    
Var3:Var2                  0.17222    0.51119     0.51240   0.336 0.736796    
Var8:Var2                 -0.01018    0.07237     0.07260   0.140 0.888497    
Var10                     -0.01141    0.08486     0.08519   0.134 0.893453    
Var10:Var7                 0.02179    0.11146     0.11162   0.195 0.845195 

> as.data.frame(importance(full.model.avg))
                   importance(full.model.avg)
Var1                               1.00000000
Var5                               1.00000000
Var3                               1.00000000
Var4                               1.00000000
Var1:Var3                          1.00000000
Var5:Var4                          1.00000000
Var3:Var4                          1.00000000
Var8                               0.73030645
Var1:Var8                          0.73030645
Var2                               0.70616912
Var7                               0.49817802
Var9                               0.26969355
Var1:Var9                          0.26969355
Var3:Var2                          0.18443111
Var10                              0.15009082
Var8:Var2                          0.05819205
Var10:Var7                         0.05183653  

Answer 1

First, you need to be very cautious about using an automated way to select models. The very name "dredge" has negative connotations among statisticians.This is generally not a good idea, for many reasons. Even the package author cautions against it in a Note at the end of the manual page for dredge():

Users should keep in mind the hazards that a “thoughtless approach” of evaluating all possible models poses. Although this procedure is in certain cases useful and justified, it may result in selecting a spurious “best” model, due to the model selection bias. “Let the computer find out” is a poor strategy and usually reflects the fact that the researcher did not bother to think clearly about the problem of interest and its scientific setting (Burnham and Anderson, 2002).

Although the model-averaging approach has advantages over choosing the one best model from a set of models, it still runs a risk of resulting in a model that depends too much on the quirks of the particular data set at hand. If you are going to use this approach, consider repeating the analysis on multiple bootstrapped samples of the data set to see how stable the results of the model selection and averaging are.

The answer to your question is that the p-values and importances reported by this package represent different types of averages.

Weights are assigned to each of the selected models (e.g., via dredge()), with the default I understand to be weighting by the relative likelihoods of the models versus the best model, as assessed by the Akaike Information Criterion. You now have a list of models, each with its included predictors variables and included interaction terms, and with its own model weight.

In a full model average as you used, the coefficient value for an individual predictor or interaction term is set to 0 in all the models from which it is omitted. Then for each predictor and interaction term, all its coefficient values and associated coefficient variances are combined in an average weighted by the corresponding model weight. The confidence intervals and p-values for each weighted-averaged coefficient are based on a normal distribution.*

The function importance() is another name for the sw() function, which reports the "Sum of model weights over all models including each explanatory variable," according to the manual page. From your output it seems to normalize this sum of weights by the sum of all model weights. So this is a different type of average: for each predictor or interaction, it's based on the weights of all the models that happened to include it regardless of the predictor's significance in the model.

It seems that all the "significant" predictors and interactions in the averaged model had importance of 1 on that latter basis, but there are several with high importance but high p-values in the averaged model. It would seem that those terms were included in all the models retained by dredge() even though they were typically not significant on their own in many of them.

It's possible that their importance has to do with their helping to bring out the significance of other predictors. You would have to examine the individual models and weights to say anything more specific. Some evaluation of why some "insignificant" predictors nevertheless were apparently retained in all the models might say something interesting about the nature of what you're examining.

As to what is best to report, why not report both? They represent different things. I would further recommend reporting at least the number of models that were retained versus the number that were evaluated by dredge(), and report in your methods the criteria by which models were selected and averaged. Although reading the software manual might make that clear, your readers should be able to get a general idea of the approach from your own presentation.

---

*Details can be found in the R code, in particular for the par.avg() function.