12
$\begingroup$

My question is whether I can use an effect size $X$ as a dependent variable and another effect size $Y$ as the independent variable in a meta-regression?

For example, I conducted a meta-analysis for the effects of exercise in drinking problems and I found significant results and high heterogeneity. I want to do a meta-regression and to use the effect size of those interventions in anxiety as an independent variable and the effect size of drinking problems as a dependent variable (assuming that each study assessed both anxiety and drinking problems and I calculated the effect sizes as Hedges's $g$).

Does this make sense to you?

$\endgroup$
1
  • $\begingroup$ My only concern would be whether you would need to transform the effect size prior to the regression...similar to transforming r using Fisher's z transform. $\endgroup$ – ReliableResearch May 6 '13 at 22:15
13
$\begingroup$

Answering this (good) question responsibly probably requires addressing meta-analysis topics beyond conventional meta-regression. I've encountered this issue in consulting clients' meta-analyses but haven't yet found or developed a satisfactory solution, so this answer isn't definitive. Below I mention five relevant ideas with selected reference citations.

First, I'll introduce terminology and notation for clarification. I assume you have paired effect-size (ES) data from $k$ independent studies, such as Study $i$'s ES estimates $y_{Di}$ for drinking problems (DP) and $y_{Ai}$ for anxiety, $i = 1, 2, \ldots, k$, as well as each estimate's conditional/sampling variance (i.e., squared standard error), say $v_{Di}$ and $v_{Ai}$. Let's denote Study $i$'s two ES parameters (i.e., true or infinite-sample ESs) as $\theta_{Di}$ and $\theta_{Ai}$. Taking the traditional random-effects view that these ES parameters vary randomly among studies, we could denote their between-studies means and variances as $\mu_D = \mathrm{E}(\theta_{Di})$ and $\tau_D^2 = \mathrm{Var}(\theta_{Di})$ for DP and as $\mu_A = \mathrm{E}(\theta_{Ai})$ and $\tau_A^2 = \mathrm{Var}(\theta_{Ai})$ for anxiety. In a conventional meta-analysis for each of DP and anxiety separately (e.g., with precisions as weights), we might assume each ES estimate's sampling distribution is normal with known variance—that is, $y_{Di} | \theta_{Di} \sim \mathcal{N}(\theta_{Di}, v_{Di})$ and $y_{Ai} | \theta_{Ai} \sim \mathcal{N}(\theta_{Ai}, v_{Ai})$ with $v_{Di}$ and $v_{Ai}$ known—at least for large within-study samples.

We don't necessarily need to take a random-effects view of this problem, but we should permit both $\theta_{Di}$ and $\theta_{Ai}$ to vary among studies for questions about their association to make sense. We might be able to do this in a heterogeneous fixed-effects framework as well, if we're careful about procedures and interpretation (e.g., Bonett, 2009). Also, I don't know whether your ESs are correlations, (standardized) mean differences, (log) odds ratios, or another measure, but the ES metric doesn't matter much for most of what I say below.

Now, on to the five ideas.

1. Ecological Bias: Assessing an association between your two ESs addresses a study-level question, not a subject-level question. I've seen meta-analysts inappropriately interpret a positive association between two ESs like yours as follows: Subjects for whom the intervention decreases anxiety more tend to decrease more on DP. Analyses of study-level ES data don't support statements like that; this has to do with ecological bias or the ecological fallacy (e.g., Berlin et al., 2002; McIntosh, 1996). Incidentally, if you had individual patient/participant data (IPD) from the studies or certain additional sample estimates (e.g., each group's correlation between anxiety and DP), then you could address certain subject-level questions about moderation or mediation involving the intervention, anxiety, and DP, such as the intervention's effect on the anxiety-DP association, or the intervention's indirect effect on DP via anxiety (e.g., intervention $\rightarrow$ anxiety $\rightarrow$ DP).

2. Meta-Regression Problems: Although you could regress $y_{Di}$ on $y_{Ai}$ using a conventional meta-regression procedure that treats $y_{Ai}$ as a fixed, known covariate/regressor/predictor, that's probably not entirely appropriate. To understand potential problems with this, consider what we might do instead if it were possible: Regress $\theta_{Di}$ on $\theta_{Ai}$ using ordinary regression (e.g., OLS) to estimate or test whether or how $\theta_{Di}$'s mean covaries with $\theta_{Ai}$. If we had each study's $\theta_{Ai}$, then using conventional meta-regression to regress $y_{Di}$ on $\theta_{Ai}$ would give us what we want, because the (simple) between-studies model is $\theta_{Di} = \beta_0 + \beta_1\theta_{Ai} + u_i$, where $u_i$ is random error. Using the same approach to regress $y_{Di}$ on $y_{Ai}$, however, ignores two problems: $y_{Ai}$ differs from $\theta_{Ai}$ due to sampling error (e.g., quantified by $v_{Ai}$) and has a within-study correlation with $y_{Di}$ due to the subject-level correlation between anxiety and DP. I suspect one or both of those problems would distort the estimate of association between $\theta_{Di}$ and $\theta_{Ai}$, such as due to regression dilution/attenuation bias.

3. Baseline Risk: Several authors have addressed problems analogous to those in #2 for meta-analyses of an intervention's effect on a binary outcome. In such meta-analyses, there's often a concern that the treatment effect covaries with the outcome's probability or rate in an untreated population (e.g., larger effect for subjects at higher risk). It's tempting to use conventional meta-regression to predict the treatment effect from a control group's risk or event rate, since the latter represents the underlying/population/baseline risk. Several authors, however, have demonstrated limitations of this simple strategy or proposed alternative techniques (e.g., Dohoo et al., 2007; Ghidey et al., 2007; Schmid et al., 1998). Some of those techniques might be suitable for or adaptable to your situation involving two multiple-endpoint ESs.

4. Bivariate Meta-Analysis: You might treat this as a bivariate problem, where Study $i$'s pair $y_i = [y_{Di}, y_{Ai}]$ is an estimate of $\theta_i = [\theta_{Di}, \theta_{Ai}]$ with conditional covariance matrix $\mathbf{V}_i = [v_{Di}, v_{\mathrm{DA}i}; v_{\mathrm{AD}i}, v_{Ai}]$—here commas separate columns and a semi-colon separates rows. We could, in principle, use bivariate random-effects meta-analysis to estimate $\mu = [\mu_D, \mu_A]$ and the between-studies covariance-component matrix $\mathbf{T} = [\tau_D^2, \tau_{DA}; \tau_{AD}, \tau_A^2]$. This could be done even if some studies contribute only $y_{Di}$ or only $y_{Ai}$ (e.g., Jackson et al., 2010; White, 2011). Besides $\tau_{DA} = \tau_{AD}$, you could also estimate other measures of the association between anxiety and DP as functions of $\mu$ and $\mathbf{T}$, such as the correlation between $\theta_{Di}$ and $\theta_{Ai}$, or the $\theta_{Di}$-on-$\theta_{Ai}$ regression slope. I'm unsure, however, how best to make inferences about any such measure of the anxiety-DP association: Do we treat both $\theta_{Di}$ and $\theta_{Ai}$ as random, or is $\theta_{Ai}$ best treated as fixed (as we might if regressing $\theta_{Di}$ on $\theta_{Ai}$), and what procedures are best for tests, confidence intervals, or other inferential results (e.g., delta method, bootstrap, profile likelihood)? Unhappily, computing the conditional covariance $v_{\mathrm{DA}i} = v_{\mathrm{AD}i}$ may be difficult, because it depends on the rarely reported within-group association between anxiety and DP; I won't address here strategies for handling this (e.g., Riley et al., 2010).

5. SEM for Meta-Analysis: Some of Mike Cheung's work on formulating meta-analytic models as structural equation models (SEMs) might offer a solution. He's proposed ways to implement a wide variety of uni- and multivariate fixed-, random-, and mixed-effects meta-analysis models using SEM software, and he provides software for this:

http://courses.nus.edu.sg/course/psycwlm/internet/metaSEM/index.html

In particular, Cheung (2009) included an example in which one ES is treated as a mediator between a study-level covariate and another ES, which is more complex than your situation of predicting one ES with another.

References

Berlin, J. A., Santanna, J., Schmid, C. H., Szczech, L. A., & Feldman, H. I. (2002). Individual patient- versus group-level data meta-regressions for the investigation of treatment effect modifiers: Ecological bias rears its ugly head. Statistics in Medicine, 21, 371-387. doi:10.1002/sim.1023

Bonett, D. G. (2009). Meta-analytic interval estimation for standardized and unstandardized mean differences. Psychological Methods, 14, 225–238. doi:10.1037/a0016619

Cheung, M. W.-L. (2009, May). Modeling multivariate effect sizes with structural equation models. In A. R. Hafdahl (Chair), Advances in meta-analysis for multivariable linear models. Invited symposium presented at the meeting of the Association for Psychological Science, San Francisco, CA.

Dohoo, I., Stryhn, H., & Sanchez, J. (2007). Evaluation of underlying risk as a source of heterogeneity in meta-analyses: A simulation study of Bayesian and frequentist implementations of three models. Preventive Veterinary Medicine, 81, 38-55. doi:10.1016/j.prevetmed.2007.04.010

Ghidey, W., Lesaffre, E., & Stijnen, T. (2007). Semi-parametric modelling of the distribution of the baseline risk in meta-analysis. Statistics in Medicine, 26, 5434-5444. doi:10.1002/sim.3066

Jackson, D., White, I. R., & Thompson, S. G. (2010). Extending DerSimonian and Laird's methodology to perform multivariate random effects meta-analyses. Statistics in Medicine, 29, 1282-1297. doi:10.1002/sim.3602

McIntosh, M. W. (1996). Controlling for an ecological parameter in meta-analyses and hierarchical models (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 9631547)

Riley, R. D., Thompson, J. R., & Abrams, K. R. (2008). An alternative model for bivariate random-effects meta-analysis when the within-study correlations are unknown. Biostatistics, 9, 172-186. doi:10.1093/biostatistics/kxm023

Schmid, C. H., Lau, J., McIntosh, M. W., & Cappelleri, J.C. (1998). An empirical study of the effect of the control rate as a predictor of treatment efficacy in meta-analysis of clinical trials. Statistics in Medicine, 17, 1923-1942. doi:10.1002/(SICI)1097-0258(19980915)17:17<1923::AID-SIM874>3.0.CO;2-6

White, I. R. (2011). Multivariate random-effects meta-regression: Updates to mvmeta. Stata Journal, 11, 255-270.

$\endgroup$
11
$\begingroup$

Built on Adam's answers, I have a few elaborations. First and most important, it is not easy to conceptualize substantive theories on how and why one effect size predicts another effect size. A multivariate meta-analysis is usually sufficient to explain the association among the effect sizes. If you are interested in hypothesizing directions among the effect sizes, you may be interested in the work by William Shadish (Shadish, 1992, 1996; Shadish & Sweeney, 1991).

As Adam has mentioned, there are some issues in applying meta-regression among the effect sizes. The main problem is that the effect sizes are conditionally distributed with known variances (and covariances). A structural equation modeling (SEM) approach may be used to address this issue (Cheung, 2008, 2013, in press). We may formulate the "true" effect sizes, $\theta_{Di}$ and $\theta_{Ai}$ in Adam's notation, as latent variables. The observed effect sizes are indicators of the "true" effect sizes:

$y_{Di} = \theta_{Di} + e_{Di}$ with $\mathrm{Var}(e_{Di})=v_{Di}$ and

$y_{Ai} = \theta_{Ai} + e_{Ai}$ with $\mathrm{Var}(e_{Ai})=v_{Ai}$.

Once we have formulated this part (the so-called measurement model), a structural model can be easily fitted among the "true" effect sizes:

$\theta_{Di} = \beta_0 + \beta_1 \theta_{Ai} + u_{Di}$,

where $\mathrm{Var}(u_{Di}) = \tau^2_{Di}$ is the residual heterogeneity of $\theta_{Di}$ and $\mathrm{Var}(\theta_{Ai})=\tau^2_{Ai}$ is the variance of $\theta_{Ai}$.

Since $y_{Di}$ and $y_{Ai}$ are conditionally correlated with a value of $v_{DAi}$, the last step is to include this conditional covariance in the model. The proposed model is: proposed model

Using the conventional SEM notation, the circles and the squares represent the latent and the observed variables. The triangle represents the intercept (or the mean).

Since the sampling variances and covariances are known in a meta-analysis, most SEM packages cannot be used to fit this model. I use the OpenMx package implemented in R to fit this model. If you want to use Mplus, you need to do some tricks to handle the known sampling variances and covariances (see Cheung, in press_a for an example).

The following example demonstrates how to fit the model with "lifecon" as the predictor and "lifesat" as the dependent variables in R. Their corresponding latent variables are called "latcon" and "latsat". The dataset is available in the metaSEM package http://courses.nus.edu.sg/course/psycwlm/Internet/metaSEM/

## Load the library with the data set  
library(metaSEM)
## OpenMx is loaded automatically after loading metaSEM
## library(OpenMx)

## Select the sample effect sizes and their sampling covariance matrix
my.df <- wvs94a[, 2:6]

## It uses the reticular action model (RAM) specification
## A matrix specifies the asymmetric paths (regression coefficients and factor loadings)
## S matrix specifies the symmetric covariances and variances
## F matrix specifies a selection matrix to select the observed variables   
lat <- mxModel("LifesatOnLifeCon",
               mxData(observed=my.df, type="raw"),
               mxMatrix(type="Full", nrow=4, ncol=4,
                        free=c(F, T, rep(F, 14)),
                        values=c(0, 0.1, 1, rep(0,4), 1, rep(0,8)),
                        labels=c(NA, "beta1", rep(NA, 14)),
                        name="A"),
               mxMatrix(type="Symm", nrow=4, ncol=4,
                        values=0, free=c(T,rep(F,3),T,rep(F,5)),
                        labels=c("Var(LifeCon)",rep(NA,3),"Var(LifeSatError)",rep(NA,2),
                                 "data.lifecon_var", "data.inter_cov", "data.lifesat_var"),                        
                        name="S"),            
               mxMatrix(type="Full", nrow=2, ncol=4,
                        values=c(rep(0,4),1,0,0,1), name="F"),
               mxMatrix(type="Full", nrow=1, ncol=4, free=c(T, T, F, F),
                        values=c(0, 0, 0, 0), labels=c("MeanLifeCon", "beta0", NA, NA), name="M"),
               mxExpectationRAM("A", "S", "F", "M", dimnames=c("latcon", "latsat", "lifecon","lifesat")),
                                mxFitFunctionML())

summary(mxRun(lat))

The output is: Summary of LifesatOnLifeCon

free parameters:
               name matrix row    col     Estimate   Std.Error
1             beta1      A   2      1  0.467619431 0.148202854
2      Var(LifeCon)      S   1      1  0.008413600 0.002537270
3 Var(LifeSatError)      S   2      2  0.002887461 0.001281026
4       MeanLifeCon      M   1 latcon  0.068825735 0.016819615
5             beta0      M   1 latsat -0.030834413 0.015565501

observed statistics:  84 
estimated parameters:  5 
degrees of freedom:  79 
-2 log likelihood:  -161.9216 
number of observations:  42 
Information Criteria: 
      |  df Penalty  |  Parameters Penalty  |  Sample-Size Adjusted
AIC:      -319.9216              -151.9216                       NA
BIC:      -457.1975              -143.2332                -158.8909
Some of your fit indices are missing.
  To get them, fit saturated and independence models, and include them with
  summary(yourModel, SaturatedLikelihood=..., IndependenceLikelihood=...). 
timestamp: 2015-01-20 18:56:09 
Wall clock time (HH:MM:SS.hh): 00:00:00.13 
optimizer:  NPSOL 
OpenMx version number: 2.0.0.4004 
Need help?  See help(mxSummary) 

As a final note, the above model is equivalent to the bivariate meta-analysis by changing the path $\beta_1$ to a double arrow $\tau^2_{DA}$ representing the covariance between the "true" effect sizes. The bivariate meta-analysis can be conducted by:

library(metaSEM)
summary( meta(y=cbind(lifesat, lifecon),
              v=cbind(lifesat_var, inter_cov, lifecon_var), 
              data=wvs94a) )

The output is:

Call:
meta(y = cbind(lifesat, lifecon), v = cbind(lifesat_var, inter_cov, 
    lifecon_var), data = wvs94a)

95% confidence intervals: z statistic approximation
Coefficients:
              Estimate   Std.Error      lbound      ubound z value  Pr(>|z|)
Intercept1  0.00134985  0.01385628 -0.02580797  0.02850766  0.0974 0.9223946
Intercept2  0.06882574  0.01681962  0.03585990  0.10179159  4.0920 4.277e-05
Tau2_1_1    0.00472726  0.00176156  0.00127465  0.00817986  2.6836 0.0072844
Tau2_2_1    0.00393437  0.00168706  0.00062779  0.00724094  2.3321 0.0196962
Tau2_2_2    0.00841361  0.00253727  0.00344064  0.01338657  3.3160 0.0009131

Intercept1    
Intercept2 ***
Tau2_1_1   ** 
Tau2_2_1   *  
Tau2_2_2   ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Q statistic on the homogeneity of effect sizes: 250.0303
Degrees of freedom of the Q statistic: 82
P value of the Q statistic: 0

Heterogeneity indices (based on the estimated Tau2):
                             Estimate
Intercept1: I2 (Q statistic)   0.6129
Intercept2: I2 (Q statistic)   0.7345

Number of studies (or clusters): 42
Number of observed statistics: 84
Number of estimated parameters: 5
Degrees of freedom: 79
-2 log likelihood: -161.9216 
OpenMx status1: 0 ("0" or "1": The optimization is considered fine.
Other values indicate problems.)

When we compare the -2 log likelihoods of these two models, they are exactly the same (-161.9216). In this case we do not gain additional insights by fitting a meta-regression on the effect sizes--a bivariate meta-analysis is already sufficient.

References

Cheung, M. W.-L. (2008). A model for integrating fixed-, random-, and mixed-effects meta-analyses into structural equation modeling. Psychological Methods, 13(3), 182–202. doi:10.1037/a0013163

Cheung, M. W.-L. (2013). Multivariate meta-analysis as structural equation models. Structural Equation Modeling: A Multidisciplinary Journal, 20(3), 429–454. doi:10.1080/10705511.2013.797827

Cheung, M. W.-L. (2014). Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach. Psychological Methods, 19(2), 211-29. doi: 10.1037/a0032968.

Shadish, W. R. (1992). Do family and marital psychotherapies change what people do? A meta-analysis of behavioral outcomes. In T. D. Cook, H. Cooper, D. S. Cordray, H. Hartmann, L. V. Hedges, R. J. Light, T. A. Louis, & F. Mosteller (Eds), Meta-analysis for explanation: A casebook (129-208). New York: Russell Sage Foundation.

Shadish, W. R. (1996). Meta-analysis and the exploration of causal mediating processes: A primer of examples, methods, and issues. Psychological Methods, 1, 47-65.

Shadish, W. R., & Sweeney, R. (1991). Mediators and moderators in meta-analysis: There's a reason we don't let dodo birds tell us which psychotherapies should have prizes. Journal of Consulting and Clinical Psychology, 59, 883-893.

$\endgroup$
3
  • $\begingroup$ Welcome to our site, Mike, and thank you very much for making this contribution. $\endgroup$ – whuber May 9 '13 at 3:25
  • $\begingroup$ Thanks, @Mike, for noting Shadish's important work and elaborating on my point #5. As you suggest, I think your 5-parameter structural model is equivalent to $\mu$ and $\mathbf{T}$ in my point #4 (e.g., $\beta_1 = \tau_{DA} / \tau_A^2$, $\beta_0 = \mu_D - \beta_1\mu_A$); interpreting $\beta_0$ and $\beta_1$ may be easier. It's interesting to consider other mean-and-covariance structures for $\mu$ and $\mathbf{T}$, especially with more effect-size parameters, but interpreting them in terms of real-world phenomena may be challenging. $\endgroup$ – Adam Hafdahl May 9 '13 at 15:03
  • $\begingroup$ About how and why one effect size predicts another effect size. Think of whatever ES and mean age (and sd) for each study. Several times I have been asked to "adjust" for age in a meta regression. $\endgroup$ – Nicolas Molano Aug 11 '20 at 12:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.