0
$\begingroup$

I have access to about ten almost identical studies from ten different locations. They have been done different years, but with the same methodology.

Each site was analysed and reported separately using the multiple linear regression model looking like: $$y=\beta_0+\beta_1X_1+\beta_2X_2+\beta_3Z_1$$ where the Xs are categorical variables (2 levels) and Z is a continous variable.

So, what I am asked to do is to perform a meta analysis for $X_1$ to get the overall "effect" of that for all the ten sites. The question is something like: Is there an overall positive effect, negative effect or no effect of $X_1$ on $Y$.

A paper that I use as a source did a similar thing by using hedges´d (or cohens´d) for each site, and then formed a confidence interval of these effect size estimates. However, that paper used one way ANOVA, thus no control variables. For a linear regression, I have never seen Cohens'd used. Don't know if that's possible or reasonable, if so.

What I have done is to derive the standardized coefficents for all $\beta_1$s, by scaling all variables to unit standard deviation and zero mean. Then I formed confidence interval based on that. Standardized coefficents should be an "effect size" since it has standarized.

However, I got criticized for not using cohens'd because it is "statistically robust". I am not sure this person understands the differences between his simple setup and my linear regression.

Any tips on how to think about this?

Could another way, since I have all the raw data, be:

  1. Add all ten datasets to one
  2. run the same model as described above, but with site as random effect (a linear mixed model)
  3. Examine this, single, estimate of $\beta_1$. If significant, there is evidence of an "overall" relation between $X_1$ and $Y$, taking the other variables inte account.

Any ideas?

$\endgroup$

1 Answer 1

1
$\begingroup$

There is no need for Cohen's d here. If all of the studies measured $y$ and $Z$ in the same way (and the two dummy variables are defined in the same way across studies), then you can just meta-analyze the raw regression coefficient for $X_1$. There is no need to standardize the variables. A raw regression coefficient is also an 'effect size'. So, take the estimated regression coefficients and their standard errors (or the squared SEs, depending on what kind of input the software expects) and feed that into your software of choice for conducting a meta-analysis.

You can also use the raw data as you propose. This is essentially an 'individual patient/participant data meta-analysis' (IPDMA). You would want to add random effects for study (which essentially means a random intercept term) and also random effects for the regression coefficients (i.e., random slopes). You also may want to allow the error variance to differ across studies (since that automatically happens when you fit 10 different regression models in the former approach, but you would have to explicitly ask for this when fitting a single mixed-effects model).

The former approach is a 'two-stage' analysis, while the latter is a 'one-stage' approach. They will give similar results as long as the two approaches are based on compatible model assumptions (for example, as noted above, by allowing the error variance to differ across studies in the mixed-effects model).

An illustration of these two different approaches is given here:

https://www.metafor-project.org/doku.php/tips:two_stage_analysis

Side-note: In this example, both approaches assume that the error variance is the same across 'studies' (in this case 'study = subject'), since lmList() by default computes a pooled error variance across the regression models.

A very nice article that contrasts these two different approaches is:

Burke, D. L., Ensor, J. & Riley, R. D. (2017). Meta-analysis using individual participant data: One-stage and two-stage approaches, and why they may differ. Statistics in Medicine, 36(5), 855-875. https://doi.org/10.1002/sim.7141

$\endgroup$
1
  • 1
    $\begingroup$ Thank you for a great answer! It feels much clearer now on how I can approach this. I am not so familiar with meta-analysis. I will disregard Cohens's and go for one of these approaches. I will begin with meta-analyse the individual regressions. Might try the metafor-package in r, that is used in the link u posted. $\endgroup$
    – Yung Gud
    Commented Sep 6, 2022 at 11:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.