Background and Problem

I have a question concerning a meta-analysis combining effects from between- and within-subject designs using log-odds ratios (OR) as the metric of interest. I am familiar with conducting meta-analyses and will be undertaking my calculations in R (using the metafor and lme4 packages). To provide greater context, the studies in question ask research subjects to make a binary decision with respect to a personal preference across one of two conditions. In some cases, each participant is assigned to a single condition (making only a single binary response); in others, each subject takes part in both conditions (making two binary responses). For now, presume I have the raw data in all cases. The issue I face is how best to calculate an OR that is comparable across design and whether I should take the correlation between conditions into account for the within-subject designs.

My Current Approach

I presently use logistic regression to estimate the OR for between-subject designs. The slope represents the OR and the sampling variance can be calculated by squaring the SE of the slope coefficient. Using this approach produces estimates comparable to equations reported in common texts such The Handbook of Research Synthesis and Meta-Analysis, 2nd Edition (p. 243). I then extend this approach to use a multilevel logistic regression model including a random intercept by subject to estimate the OR for within-subject designs while account for the dependency between conditions. The OR and sampling variance are otherwise calculated in the same fashion.

My Questions:

With this in mind, I would like to ask:

  1. Is it reasonable to meta-analytically aggregate OR calculated using standard and multilevel logistic regression?
  2. Would it be better to use standard logistic regression for both designs (ignoring the correlation between conditions for the within-subject designs)?
  • $\begingroup$ This is a good question, but you may want to frame it in a more software-neutral way. Questions that are just about how to use R are generally off topic & typically closed. $\endgroup$ Mar 28, 2016 at 17:40
  • $\begingroup$ Thank you for the tip! This is my first time posting, so I am still learning the ropes. I will edit the framing to make it more software neutral this evening or tomorrow afternoon. $\endgroup$
    – jmfawcet
    Mar 28, 2016 at 18:56

2 Answers 2


I'll focus in my answer purely on the question on how to compute a (log) OR based on a within-subjects design that is comparable to that from a between-subjects design.

Suppose you have a within-subjects design with these data:

                         decision1   decision2   total
condition1   decision1   s           t           a
             decision2   u           v           b
             total       c           d           n

Note that this is the 'paired-subjects' 2x2 table based on n subjects. This table can be rearranged into a 'between-subjects' 2x2 table:

             decision1   decision2   total
condition1   a           b           n
condition2   c           d           n

Then you can compute what is often called the "marginal OR" with the usual equation for computing an odds ratio with:

$$OR = \frac{ad}{bc}$$

And for meta-analytic purposes, we usually work with the log(OR), so just take the log of that. This value is then comparable to that obtained from a between-subjects design.

However, note that the same n subjects are used to compute the cell entries under condition1 and condition2, so the data are not independent. This needs to be taken into consideration when computing the sampling variance of the marginal log odds ratio. Based on Becker and Balagtas (1993) (see also: Elbourne et al., 2002, and Stedman et al., 2011), we can compute (or to be precise: estimate) the sampling variance of the marginal log(OR) with:

$$Var(log[OR]) = \frac{1}{a} + \frac{1}{b} + \frac{1}{c} + \frac{1}{d} - \frac{2\Delta}{n},$$


$$\Delta = n^2 \left(\frac{ns - ac}{abcd}\right).$$

(Recall that $s$ is the upper-left cell count from the paired-subjects table.)


Becker, M. P., & Balagtas, C. C. (1993). Marginal modeling of binary cross-over data. Biometrics, 49(4), 997-1009.

Elbourne, D. R., Altman, D. G., Higgins, J. P. T., Curtin, F., Worthington, H. V., & Vail, A. (2002). Meta-analyses involving cross-over trials: Methodological issues. International Journal of Epidemiology, 31(1), 140-149.

Stedman, M. R., Curtin, F., Elbourne, D. R., Kesselheim, A. S., & Brookhart, M. A. (2011). Meta-analyses involving cross-over trials: Methodological issues. International Journal of Epidemiology, 40(6), 1732-1734.

  • 1
    $\begingroup$ @jmfawcet There is a minor error in the equation given by Stedman et al. (2011). It should be $-2\Delta/n$, not $-\Delta/(2n)$. $\endgroup$
    – Wolfgang
    May 4, 2016 at 16:54
  • $\begingroup$ There is now a follow-up question to this one that I imagine you might also be able to answer if you have time. You can find it here. $\endgroup$
    – jmfawcet
    Nov 30, 2016 at 4:18

You have the individual participant data so you have two choices (a) a one step approach which since you use R can be implemented in lme4 or (b) a two-step approach where you reduce each study to a single summary statistic and then meta-analyse them using, as you suggest metafor. If you do the single step approach you need to specify study and subject as random effects. Each subject in your between subject designs then forms his/her own cluster in the design with a single observation whereas the other clusters from the within designs have two observations per cluster. The one-step design of course lends itself to the inclusion of person level covariates if you have them.

  • $\begingroup$ Thank you for this answer. Just to clarify, I was aware that a multi-level analysis using the combined raw data was an option (although, see link) – however, what I am hoping to learn is how to best calculate a within-subject OR such that it would be congruent with a more typical between-subject OR. So, really, this question pertains to option (b) from response. Could you expand upon that component of your answer? $\endgroup$
    – jmfawcet
    Apr 8, 2016 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.