5
$\begingroup$

I have data from 40 people, each measured once in 2 conditions. For reasons tangential to my question I'm not comfortable using parametric analyses to compare the two conditions, so I'm employing bootstrapping. However, when I run a bootstrap analysis (resampling people with replacement, computing condition means, then repeating 1000 times), the condition means I obtain appear to be correlated across iterations. I noticed this because when I plot the 95% confidence interval in each condition the intervals capture each others' means, but when I compute a difference score between the conditions means within each iteration, the 95% confidence interval of this difference score excludes zero. Going back to the results of the bootstrapping, I find that the correlation of the conditions across the bootstrap iterations is .7! Should this affect my confidence in the interval generated for the difference score? Alternatively, does this tell me something (possibly about individual differences) about my data otherwise?

$\endgroup$
  • $\begingroup$ Assuming you've got a valid rationale for comparing the same people in the two conditions without a true control group, I'm not sure the correlation across bootstrap iterations is really something to be worried about in itself. It seems like you are, in effect, calculating CIs for the Average Treatment Effect on the Treated (no Wikipedia entry yet, sorry) and interpreting that as a meaningful outcome of interest. That said, I'm having a hard time following exactly what steps you took to estimate the differences between conditions. Is it a simple difference of means? Something else? $\endgroup$ – ashaw Feb 27 '11 at 22:17
  • $\begingroup$ @ashaw: Comparison of conditions experienced by the same individuals is a standard methodological approach that increases power by permitting removal of between-Ss variance that usually obscures effect variance in completely between-Ss designs. Regarding the computation of the difference score CI, I have edited the post above to clarify that on each iteration the condition means were computed and collapsed to a difference, yielding a distribution of differences from which the 95% interval. $\endgroup$ – Mike Lawrence Feb 28 '11 at 18:19
  • $\begingroup$ that helps for sure. Apologies if you read the first phrase of my comment to suggest that the approach was invalid in any way. Rather, I meant to point out that the validity of my response was contingent on the validity of the study design. In any event, now I can mull over a real response a little more... $\endgroup$ – ashaw Feb 28 '11 at 18:52
1
$\begingroup$

While you didn't say anything about the actual condtions (experimental treatment), think of the following example: you measure the weight of 40 people at some point during the day, then set them the task to drink (as much as possible of) 5 quarts of water (experimental manipulation), and them weigh them again (hinting at "yes" being the answer to your final question...).

Now, the values for the means of each group will, pre and post treatment, very likely overlap (given that, unless you select people with the same pre-treatment weight, the general population shows a great variability). The mean of the difference between post and pre (post - pre), on the other hand, is likely to be significantly different from (i.e. greater than) 0.

In other words, without knowing a little more about your conditions (e.g. how does the metric change between conditions? do you expect it to show signs of stability?) and also about the variability on that measure across the population (if you think of the measure as a random variable, each subject has his/her own probability distribution of getting a certain score under a certain condition; and the measurement error might be very small compared to the reliable difference of each participants measure compared to the population mean).

Does any of this help?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.