I have two studies with two experimental conditions each. The paradigms and measures used were the same, the studies only differed in a feature of the experimental setting (you could probably see "study" as a between-subjects factor, but I want to refrain from analyzing the two different data sets as if they came from the exact same study.) What I would like to test is if the size of the effect of the experimental condition is greater in the second study than in the first study. (In my case, the experimental condition was a within-subjects factor, but I don't think that this is conceptually important.)

To do this, I thought about a bootstrapping approach: I took 10000 random resamples of the data of the first study and, separately, of the second study, and for each resampled study data calculated the effect size of condition (standardized mean difference, Cohen's d). Then I calculated the difference of the effect sizes, thus the outcome of each resampling circle was one value, the difference between effect sizes. Finally, I calculated the confidence interval for this difference, and checked whether it included zero.

Here's the (I guess rather amateurish) R code I wrote for this:

boot.ddiff.paired = function(xa1,xa2,xb1,xb2, n = 10000) {
i = 1
out.da = numeric(0)
out.db = numeric(0)
out.ddiff = numeric(0)
mata = cbind(xa1,xa2)
matb = cbind(xb1,xb2)
while(i <= n) {
    samp.mata = mata[sample(1:nrow(mata),nrow(mata),replace = TRUE),]
    samp.matb = matb[sample(1:nrow(matb),nrow(matb),replace = TRUE),]

    mdiffa = mean(samp.mata[,2] - samp.mata[,1])
    sava = sqrt((sd(samp.mata[,2])^2 + sd(samp.mata[,1])^2)/2)
    da = mdiffa/sava

    mdiffb = mean(samp.matb[,2] - samp.matb[,1])
    savb = sqrt((sd(samp.matb[,2])^2 + sd(samp.matb[,1])^2)/2)
    db = mdiffb/savb

    ddiff = db - da

    out.da = c(out.da,da)
    out.db = c(out.db,db)
    out.ddiff = c(out.ddiff,ddiff)
    i = i+1
return(list(ES.1 = out.da, ES.2 = out.db, ES.diff = out.ddiff))


So my question is: Does this approach make any sense? One the one hand, I never heard about anybody doing something like this, on the other hand, I could imagine that the "plug-in"-principle of bootstrapping can also be applied here.

Also a general question: I'd like to "pep" the analysis up by calculating a p-value, just calculating the percentage of bootstrapped effect size differences <= 0 . This of course corresponds perfectly to the percentile confidence intervals (if the percentile intervals just include 0, the p-value is also just above 0.05). But I read that it is better to report BCa CIs. Is there a way to calculate a p-value that also matches the BCa CIs?

  • $\begingroup$ I've yet to find a scenario in which the bootstrap was invalid. It can fail in small sample sizes or with sparse strata. You could compare this answer with a two-sample t-test to confirm that your are getting similar answers - in that case I would probably prefer the t-test due to simplicity. $\endgroup$
    – alex keil
    Commented Oct 19, 2016 at 17:12
  • 1
    $\begingroup$ The p-value in a bootstrap is weird. A p-value generally quantifies the probability of observing your an answer as extreme as the one given by your data, given that the null is true. In that case, you should be resampling from the null and finding the proportion of bootstrap datasets in which you found answers as extreme as your data. The p-value from a bootstrap represents the probability of a non-null result, given your study data - this is not a traditional p-value. $\endgroup$
    – alex keil
    Commented Oct 19, 2016 at 17:15


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.