I have the following example data, stored in the variable
Study d Variance Category 1 0 0.1 A 1 5 0.1 B 1 10 0.1 C 2 20 0.1 A 2 25 0.1 B
That is, I'm missing data for Category C for Study 2.
I then fit two meta-analytic models on this data with
Category as the moderator in both cases.
Analysis1 is a two-level model model, with
Study as the random factor,
Analysis1 <- rma.mv(d, Variance, random = ~ 1 | Study, mods = ~ factor(Category) - 1, data=TheData)
which gives me the following output:
Multivariate Meta-Analysis Model (k = 5; method: REML) Variance Components: estim sqrt nlvls fixed factor sigma^2 199.9504 14.1404 2 no Study Test for Residual Heterogeneity: QE(df = 2) = 4000.0000, p-val < .0001 Test of Moderators (coefficient(s) 1,2,3): QM(df = 3) = 626.6563, p-val < .0001 Model Results: estimate se zval pval ci.lb ci.ub factor(Category)A 10.0000 10.0013 0.9999 0.3174 -9.6021 29.6021 factor(Category)B 15.0000 10.0013 1.4998 0.1337 -4.6021 34.6021 factor(Category)C 19.9975 10.0050 1.9987 0.0456 0.3880 39.6070 * --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Analysis2 is a single-level model (and therefore without any random factors),
Analysis2 <- rma.mv(d, Variance, mods = ~ factor(Category) - 1, data=TheData)
which gives me the following output:
Multivariate Meta-Analysis Model (k = 5; method: REML) Variance Components: none Test for Residual Heterogeneity: QE(df = 2) = 4000.0000, p-val < .0001 Test of Moderators (coefficient(s) 1,2,3): QM(df = 3) = 7500.0000, p-val < .0001 Model Results: estimate se zval pval ci.lb ci.ub factor(Category)A 10.0000 0.2236 44.7214 <.0001 9.5617 10.4383 *** factor(Category)B 15.0000 0.2236 67.0820 <.0001 14.5617 15.4383 *** factor(Category)C 10.0000 0.3162 31.6228 <.0001 9.3802 10.6198 *** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
As can be seen, when using the single-level model,
Analysis2, each category estimate is simply the mean of the values for that category. However, when using the three-level model,
Analysis1, the estimates for Category A and B are the same as in the single-level model, while Category C seems to be estimated based on an extrapolation of what its value would/should be in Study 2. That is, what seems to be going on (put in an extremely non-mathematical language, since I haven't fully grasped the inner workings of how these analyses are performed) is that the model goes: "Oh, there's no data for Category C in study 2. Well, Category C was larger than both A and B in Study 1 so I guess it should be larger than Category A and B in Study 2 as well."
This is probably fine for a lot of cases, but are there situations when it's simply not appropritate to let the model extrapolate in this way? For example, I have a dataset where the different categories represent different categories of tests that the participants in psychological studies participated in. That is, the categories are things like verbal, visuo-spatial, and tactile, where each category merely designates the type of the test and not a specific test (for example, the category tactile could be applicable on hundereds of different tests). Further, all categories are never represented within the same study, so the data is just filled with missing data.
Here, it seems quite strange to extrapolate when trying to estimate the actual values of each category, especially since I have so many different categories as well (is it realistic to think that the extrapolation would be any good when I, for example, only have 2 out of 9 possible categories represented for a given study?). It simply seems messy to apply this method on this type of data. Am I on to something here?