2
$\begingroup$

I am performing a meta-analysis in R with metafor package. Primary studies report the reduction of plant species to certain amount of nitrogen fertilization. Effect size is calculated as the ratio of number of species in the treatment group divided the control for each observation. Following Borenstein (2009), there is no reason to believe a fixed effect model should be used since these experiments are all undertaken in different ecosystems, vegetation types, soil type et cetera. However, heterogeneity of the pooled effect sizes obtained using random models of rma() is extremely low.

Total heterogeneity is very low (I^2 = 0.00%) for regressions fitted with "DL" method:

> res_means <- rma(S_ratio, Variance, data = dat_S_ratio, method="DL")
> res_means

Random-Effects Model (k = 236; tau^2 estimator: DL)

tau^2 (estimated amount of total heterogeneity): 0 (SE = 0.0090)
tau (square root of estimated tau^2 value):      0
I^2 (total heterogeneity / total variability):   0.00%
H^2 (total variability / sampling variability):  1.00

Test for Heterogeneity: 
Q(df = 235) = 170.8544, p-val = 0.9994

Model Results:

estimate       se     zval     pval    ci.lb    ci.ub          
  0.7403   0.0203  36.5345   <.0001   0.7005   0.7800      *** 

---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Even using REML (I read somewhere that is actually more accurate), heterogeneity is still very low (I^2 = 10.99%):

> res_means <- rma(S_ratio, Variance, data = dat_S_ratio, random=~1|PrimaryStudy, control=list(stepadj=.5))
> res_means

Random-Effects Model (k = 236; tau^2 estimator: REML)

tau^2 (estimated amount of total heterogeneity): 0.0120 (SE = 0.0090)
tau (square root of estimated tau^2 value):      0.1096
I^2 (total heterogeneity / total variability):   10.99%
H^2 (total variability / sampling variability):  1.12

Test for Heterogeneity: 
Q(df = 235) = 170.8544, p-val = 0.9994

Model Results:

estimate       se     zval     pval    ci.lb    ci.ub          
  0.7628   0.0221  34.4510   <.0001   0.7194   0.8062      *** 

---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Subgroup analysis provide also very low values of heterogeneity when effect size is calculated for each subgroup.

Is it possible to have such unexpected value of heterogeneity? I suspect this is due to the way I calculated the variance but I am not sure about it. Indeed variance was not extracted from primary studies but back-calculated with the following for each observation:

Variance calculation obtained with the Delta method, assuming data are following a Poisson distribution. Sst is the richness of species in the treatment group, Ssc is the richness of species at the control group

Variance calculation obtained with the Delta method, assuming data are following a Poisson distribution. Sst is the richness of species in the treatment group, Ssc is the richness of species at the control group

Looking at the forest plot, confidence intervals looks similar to each other in terms of range (ok, there are differences but not THAT big), although for each observation the effect size is different. Is this an explanation why heterogeneity is so low?

Does it make sense to still perform a meta-regression on observation with such low heterogeneity?

$\endgroup$
  • 1
    $\begingroup$ Your outcome (S_ratio) is a count divided by another count? Then I would suggest to log transform those values. Let's say the ratios in two studies are 20/10 and 10/20. Then the average would be 1.25, which makes little sense. Ratios of counts are not symmetric around 1. Log transforming the ratios solves this issue. Also, log transforming should make the sampling distribution more normal (an assumption underlying the model you are using). The variance is then $1/S_{st} + 1/S_{sc}$. $\endgroup$ – Wolfgang Nov 3 '16 at 15:31
  • $\begingroup$ Since you are comparing between and within it may be that your studies have very high within variance. But follow @Wolfgang advice first. $\endgroup$ – mdewey Nov 3 '16 at 16:08
  • $\begingroup$ yes. Thanks a lot for the suggestion @Wolfgang. Indeed with log transformed {S_ratio} heterogeneity is higher (I^2 = 49%, with "DL" estimator). $\endgroup$ – Gabriele Midolo Nov 3 '16 at 17:35
  • $\begingroup$ The Poisson distribution carries a rather stringent assumption about the variance of your measures (notably the variance exactly equals the mean). Might this bear on the assumptions you are making in your variance estimate? $\endgroup$ – Alexis Nov 3 '16 at 21:10
  • $\begingroup$ @GabrieleMidolo can you do a sanity check and create a forest plot of effects and CIs before pooling them all together? $\endgroup$ – AdamO Feb 1 '18 at 16:37
1
$\begingroup$

So this one does not go unanswered for ever here is @Wolfgang's comment which the OP agreed solved his problem.

Your outcome (S_ratio) is a count divided by another count? Then I would suggest to log transform those values. Let's say the ratios in two studies are 20/10 and 10/20. Then the average would be 1.25, which makes little sense. Ratios of counts are not symmetric around 1. Log transforming the ratios solves this issue. Also, log transforming should make the sampling distribution more normal (an assumption underlying the model you are using). The variance is then $\frac{1}{S_{st}} + \frac{1}{S_{sc}}$

Here, as a bonus, is a reference justifying the OP's comment about using REML to estimate $\tau^2$. It is a paper by Viechtbauer entitles "Bias and efficiency of meta--analytic variance estimators in the random--effects model" available here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.