2
$\begingroup$

McFadden's pseudo-R² is a well-known coefficient of determination. If I am right, it can be calculated by 1-(mod_deviance/null_deviance), where mod_deviance is the deviance value for the fitted model and null_deviance is the deviance for the null model which includes only an intercept as predictor.

Can we calculate McFadden's pseudo-R² for averaged models? This is a non-answered question to me, for example here: Pseudo R-squared of averaged model. My case is similar - I work in R and use the MuMIn-package. My "averaged model" is obtained from the top-ranked models in the dredge function. This is: I try all possible variable combinations and those models with the best fit are averaged with the function model.avg. The summary of averaged models misses information about the model's deviance. p2R in the pscl-package yields no result. So - no chance to calculate McFadden's pseudo-R²?

A collegue of mine proposed the following. Suppose we have an averaged model with model-averaged coefficients:

Model-averaged coefficients:  
(full average) 
               Estimate Std. Error Adjusted_SE z_value Pr(>|z|)  
(Intercept)     -0.2541     0.9522      0.9538   0.266   0.7899  
A              -13.1497     8.6335      8.6969   1.512   0.1305  
B               22.0164    10.9753     11.0683   1.989   0.0467 *
C                8.8932    10.3528     10.3964   0.855   0.3923   
D                0.1664     1.4000      1.4020   0.119   0.9055  

(conditional average) 
               Estimate Std. Error Adjusted_SE z_value Pr(>|z|)  
(Intercept)     -0.2541     0.9522      0.9538   0.266   0.7899  
A              -13.9140     8.2605      8.3305   1.670   0.0949 .
B               22.0164    10.9753     11.0683   1.989   0.0467 *
C               14.8596     9.5094      9.5886   1.550   0.1212   
D                0.9010     3.1548      3.1596   0.285   0.7755  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Now, he "builds" a new model by taking the estimates of the conditional average model:

my_new_model <- glmer(presence_absence ~ offset(-13.9140*A)+ offset(22.0164*B)+
    offset(0.9010*D)+ offset(14.8596*C)+ (1|RandomFactor),
    family="binomial", data=my_data, na.action="na.fail")

In the summary of the model, we find the model's deviance:

AIC      BIC     logLik deviance df.resid 
140.6    146.7    -68.3    136.6      150 

Now, we also calculate a Null-model:

NullModel<-glmer(presence_absence ~ (1|RandomFactor), family = “binomial",
    data=my_data, na.action = "na.fail" )

In the NullModels summary we find the Null deviance:

AIC      BIC   logLik deviance df.resid 
198      204      -97      194      150 

And finally, McFadden's R²:

1-(136.6/ 194)
[1] 0.2958763

To be honest, this approach seems rather complicated to me; however, it might be useful. I fear that I am not enough a statistician to fully understand what he did there and he couldn't explain it properly to me. I wonder if his approach is valid - if so, why is it not implemented in any function/package (as far as I know...)?

I'd appreciate comments and ideas on this!

$\endgroup$
  • 1
    $\begingroup$ If what you mean by an average model is simply a model with arithmetic average of coefficients of the same parameters among top candidate models, then it is also a another model where deviance can be calculated. The methodology suggested in your question in my opinion is correct. Basically you may pretend that you came up with the average model without actually averaging, but with some other fitting algorithm and compare it to the null deviance. $\endgroup$ – Cagdas Ozgenc Mar 13 '18 at 8:48
  • $\begingroup$ Thank you for your comment, @Cagdas Ozgenc. I am not sure wheather the output of model.avg in MuMIn-package is simply a model with arithmetic average of coefficients. I will try to find out. If the methodology really is correct, then I wonder why it's not possible to obtain the deviance of an averaged model... $\endgroup$ – yenats Mar 13 '18 at 13:33
  • 1
    $\begingroup$ The coefficients are a weighted average, where weights are the IC-weights. The reason the deviance is not given is that there are several ways to calculate the averaged-predicted values, which obviously change the residuals, and hence the deviance. $\endgroup$ – Kamil Bartoń Mar 16 '18 at 15:17
  • $\begingroup$ Do you think there is a "preferable" way to calculate the average-predicted values in order to obtain a reasonable deviance-value @Kamil Bartón? As there are several ways, I suppose there is no gold standard - however, perhaps you can indicate a good starting point to calculate an averaged model's deviance. A sensible way to provide - for example - McFadden's pseudo-R² for an averaged model could be to also be transparent about the calculation of the averaged-predicted values, I suppose. $\endgroup$ – yenats Mar 19 '18 at 10:03
1
$\begingroup$

In this special case, it became a standard procedure to calculate the R² for all models used to build the final average model. In a results table, one can then show the range of these R²-values.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.