# Metafor package: Interpreting meta-regression model

I am conducting a reliability generalization for Perceived Stress Scale. This instrument has 3 version: PSS 4, PSS 10, and PSS 14. Descriptive statistics showed PSS 4 (alpha = .74, 95% CI = .70-.76) have lower alpha coefficient than PSS 10 and PSS 14 (both have identical alpha value; (alpha = .84, 95% CI = .83-.85).

I entered 9 categorical and 3 continuous moderators into the meta-regression with code:

res <- rma(measure="ABT", ai=ai, mi=mi, ni=ni, mods = ~ version + size + translation + student + psychiatry + physical + drug + pregnant + mix + male + age + sd, data=Dataset, digits=3)
res


The result is as below:

Mixed-Effects Model (k = 105; tau^2 estimator: REML)

tau^2 (estimated amount of residual heterogeneity):     0.089 (SE = 0.016)
tau (square root of estimated tau^2 value):             0.298
I^2 (residual heterogeneity / unaccounted variability): 91.35%
H^2 (unaccounted variability / sampling variability):   11.56
R^2 (amount of heterogeneity accounted for):            32.23%

Test for Residual Heterogeneity:
QE(df = 90) = 1416.665, p-val < .001

Test of Moderators (coefficient(s) 2,3,4,5,6,7,8,9,10,11,12,13,14,15):
QM(df = 14) = 56.397, p-val < .001

Model Results:

estimate     se    zval   pval   ci.lb   ci.ub
intrcpt                           1.160  0.172   6.743  <.001   0.823   1.497  ***
version[T.PSS 10]                 0.267  0.117   2.284  0.022   0.038   0.497    *
version[T.PSS 14]                 0.234  0.128   1.833  0.067  -0.016   0.485    .
size[T.201-1000]                  0.023  0.075   0.313  0.755  -0.123   0.170
size[T.>1000]                     0.261  0.140   1.861  0.063  -0.014   0.537    .
translation[T.Translated]        -0.235  0.075  -3.143  0.002  -0.381  -0.088   **
student[T.Student]               -0.097  0.097  -0.991  0.322  -0.288   0.094
psychiatry[T.Psychiatric]         0.071  0.178   0.402  0.688  -0.277   0.420
physical[T.Physical problems]     0.068  0.100   0.678  0.498  -0.128   0.263
drug[T.Drug-related]             -0.120  0.235  -0.511  0.609  -0.581   0.341
pregnant[T.Pregnant-related]      0.001  0.175   0.004  0.997  -0.342   0.343
mix[T.Mixed sample]              -0.053  0.171  -0.307  0.759  -0.388   0.283
male                             -0.001  0.001  -0.540  0.589  -0.003   0.002
age                               0.001  0.003   0.212  0.832  -0.005   0.006
sd                                0.071  0.020   3.495  <.001   0.031   0.110  ***


The result showed that the reliability of PSS 10 (b=0.267, p<.05) was statistically different from PSS 4 (reference level). Since this is a 3-level moderator, I further examine the factor with code:

anova(res, btt=2:3)


The result showed:

Test of Moderators (coefficient(s) 2,3):
QM(df = 2) = 5.215, p-val = 0.074


which suggest that the factor as a whole is not a significant predictor.

My question here is: 1. Is the analysis above methodologically sound? 2. How can i interpret this finding? Does it means that PSS version is not a significant predictor of variability in reliability? 3. Or should I perform separate analysis for each moderator? I've lost huge number of cases due to low report rate of male ratio, mean age, and SD (268 alpha estimates were extracted).

1. Yes, based on what you have shown, I would say that the analysis is sensible. One concern might be the relatively large number of moderator variables (or more specifically, model coefficients) relative to the number of estimates. Right now, you have $105 / 14 = 7.5$ estimates per coefficient (not counting the intercept). Some might want that ratio to be closer to 10 or even 15, but some might also be okay with a ratio of 5. None of these are right or wrong, but the lower the ratio, the more concerned I would be with overfitting.
2. Indeed, strictly speaking, the PSS version factor fails to be significant at $\alpha = .05$. However, I think you can still discuss this factor -- cautiously. Based on psychometric theory and all else equal, it is to be expected that longer versions would lead to higher reliability, which is indeed what you find here (although the 14-item version does not seem to yield, on average, higher reliability than the 10-item version -- maybe those 4 extra items are not as internally consistent as the rest or maybe there is something else that is different about studies examining the 14-item version that is not captured by all the other moderator variables already included in the model).