Skip to main content
Notice removed Draw attention by splint
Bounty Ended with Jacques Wainer's answer chosen by splint
added tag; formatted; edited; removed thanks
Source Link
gung - Reinstate Monica
  • 147.5k
  • 89
  • 406
  • 717

Effect size and bootstrapping in paired t test-test

I have multiple paired t$t$-tests, such as one giving results:

t(14)=2.7, p=.017$t_{14} = 2.7,\ p = .017$

althoughAlthough people seem to do effect sizes in different ways in repeated samples, I have taken the mean difference divided by the standard deviation of the differences (I'll call this d$d$, though maybe I should call it something else?) and get 0.70$0.70$. I also have a very strong correlation between the samples, not sure if that is problematic.

I would like to put confidence limits around my effect size estimate. To do so, I randomly resample from the difference scores, compute d$d$ in the same way and repeat 1000 times. My question is whether this is a good approach, rather than, say, just giving confidence limits around the unstandardised difference or resampling from the original samples. My bootstrap gives me a mean d$d$ of 0.79$0.79$ with confidence limits of [0.4, 1.4]$[0.4, 1.4]$. I've tried this on other random data too. Why am I getting a consistently higher d$d$ from bootstrapping, and why are the intervals asymmetric? Is this because of skew in the (difference) scores, and does this make this approach more or less robust?

Thanks!

 

Edit: here is an example of the data involved. 15 people were sampled atmeasured two times.

TimeA: 1999 1501 1552 2385 2488 1257 1806 1348 2048 1810 1308 2310 1247 1839 1235

MeanAMean A = 1742; SD = 435

TimeB: 2040 1601 1623 2386 2671 1218 1719 1405 2079 2017 1356 2324 1616 1878 
1370

MeanBMean B = 1820; SD = 426

Mean
Mean difference = 78, SD of differences = 111, d$d$ = 0.70

    A    B
 1999 2040
 1501 1601
 1552 1623
 2385 2386
 2488 2671
 1257 1218
 1806 1719
 1348 1405
 2048 2079
 1810 2017
 1308 1356
 2310 2324
 1247 1616
 1839 1878
 1235 1370

Effect size and bootstrapping in paired t test

I have multiple paired t-tests, such as one giving results

t(14)=2.7, p=.017

although people seem to do effect sizes in different ways in repeated samples, I have taken the mean difference divided by the standard deviation of the differences (I'll call this d, though maybe I should call it something else?) and get 0.70. I also have a very strong correlation between the samples, not sure if that is problematic.

I would like to put confidence limits around my effect size estimate. To do so, I randomly resample from the difference scores, compute d in the same way and repeat 1000 times. My question is whether this is a good approach, rather than, say, just giving confidence limits around the unstandardised difference or resampling from the original samples. My bootstrap gives me a mean d of 0.79 with confidence limits of [0.4, 1.4]. I've tried this on other random data too. Why am I getting a consistently higher d from bootstrapping, and why are the intervals asymmetric? Is this because of skew in the (difference) scores, and does this make this approach more or less robust?

Thanks!

Edit: here is an example of the data involved. 15 people were sampled at two times.

TimeA: 1999 1501 1552 2385 2488 1257 1806 1348 2048 1810 1308 2310 1247 1839 1235

MeanA = 1742; SD = 435

TimeB: 2040 1601 1623 2386 2671 1218 1719 1405 2079 2017 1356 2324 1616 1878 1370

MeanB = 1820; SD = 426

Mean difference = 78, SD of differences = 111, d = 0.70

Effect size and bootstrapping in paired t-test

I have multiple paired $t$-tests, such as one giving results:

$t_{14} = 2.7,\ p = .017$

Although people seem to do effect sizes in different ways in repeated samples, I have taken the mean difference divided by the standard deviation of the differences (I'll call this $d$, though maybe I should call it something else?) and get $0.70$. I also have a very strong correlation between the samples, not sure if that is problematic.

I would like to put confidence limits around my effect size estimate. To do so, I randomly resample from the difference scores, compute $d$ in the same way and repeat 1000 times. My question is whether this is a good approach, rather than, say, just giving confidence limits around the unstandardised difference or resampling from the original samples. My bootstrap gives me a mean $d$ of $0.79$ with confidence limits of $[0.4, 1.4]$. I've tried this on other random data too. Why am I getting a consistently higher $d$ from bootstrapping, and why are the intervals asymmetric? Is this because of skew in the (difference) scores, and does this make this approach more or less robust?

 

Edit: here is an example of the data involved. 15 people were measured two times.

Mean A = 1742; SD = 435 
Mean B = 1820; SD = 426
Mean difference = 78, SD of differences = 111, $d$ = 0.70

    A    B
 1999 2040
 1501 1601
 1552 1623
 2385 2386
 2488 2671
 1257 1218
 1806 1719
 1348 1405
 2048 2079
 1810 2017
 1308 1356
 2310 2324
 1247 1616
 1839 1878
 1235 1370
example data added
Source Link
splint
  • 93
  • 1
  • 7

I have multiple paired t-tests, such as one giving results

t(14)=2.7, p=.017

although people seem to do effect sizes in different ways in repeated samples, I have taken the mean difference divided by the standard deviation of the differences (I'll call this d, though maybe I should call it something else?) and get 0.70. I also have a very strong correlation between the samples, not sure if that is problematic.

I would like to put confidence limits around my effect size estimate. To do so, I randomly resample from the difference scores, compute d in the same way and repeat 1000 times. My question is whether this is a good approach, rather than, say, just giving confidence limits around the unstandardised difference or resampling from the original samples. My bootstrap gives me a mean d of 0.79 with confidence limits of [0.4, 1.4]. I've tried this on other random data too. Why am I getting a consistently higher d from bootstrapping, and why are the intervals asymmetric? Is this because of skew in the (difference) scores, and does this make this approach more or less robust?

Thanks!

Edit: here is an example of the data involved. 15 people were sampled at two times.

TimeA: 1999 1501 1552 2385 2488 1257 1806 1348 2048 1810 1308 2310 1247 1839 1235

MeanA = 1742; SD = 435

TimeB: 2040 1601 1623 2386 2671 1218 1719 1405 2079 2017 1356 2324 1616 1878 1370

MeanB = 1820; SD = 426

Mean difference = 78, SD of differences = 111, d = 0.70

I have multiple paired t-tests, such as one giving results

t(14)=2.7, p=.017

although people seem to do effect sizes in different ways in repeated samples, I have taken the mean difference divided by the standard deviation of the differences (I'll call this d, though maybe I should call it something else?) and get 0.70. I also have a very strong correlation between the samples, not sure if that is problematic.

I would like to put confidence limits around my effect size estimate. To do so, I randomly resample from the difference scores, compute d in the same way and repeat 1000 times. My question is whether this is a good approach, rather than, say, just giving confidence limits around the unstandardised difference or resampling from the original samples. My bootstrap gives me a mean d of 0.79 with confidence limits of [0.4, 1.4]. I've tried this on other random data too. Why am I getting a consistently higher d from bootstrapping, and why are the intervals asymmetric? Is this because of skew in the (difference) scores, and does this make this approach more or less robust?

Thanks!

I have multiple paired t-tests, such as one giving results

t(14)=2.7, p=.017

although people seem to do effect sizes in different ways in repeated samples, I have taken the mean difference divided by the standard deviation of the differences (I'll call this d, though maybe I should call it something else?) and get 0.70. I also have a very strong correlation between the samples, not sure if that is problematic.

I would like to put confidence limits around my effect size estimate. To do so, I randomly resample from the difference scores, compute d in the same way and repeat 1000 times. My question is whether this is a good approach, rather than, say, just giving confidence limits around the unstandardised difference or resampling from the original samples. My bootstrap gives me a mean d of 0.79 with confidence limits of [0.4, 1.4]. I've tried this on other random data too. Why am I getting a consistently higher d from bootstrapping, and why are the intervals asymmetric? Is this because of skew in the (difference) scores, and does this make this approach more or less robust?

Thanks!

Edit: here is an example of the data involved. 15 people were sampled at two times.

TimeA: 1999 1501 1552 2385 2488 1257 1806 1348 2048 1810 1308 2310 1247 1839 1235

MeanA = 1742; SD = 435

TimeB: 2040 1601 1623 2386 2671 1218 1719 1405 2079 2017 1356 2324 1616 1878 1370

MeanB = 1820; SD = 426

Mean difference = 78, SD of differences = 111, d = 0.70

Tweeted twitter.com/StackStats/status/672783571703197696
Notice added Draw attention by splint
Bounty Started worth 50 reputation by splint
Source Link
splint
  • 93
  • 1
  • 7

Effect size and bootstrapping in paired t test

I have multiple paired t-tests, such as one giving results

t(14)=2.7, p=.017

although people seem to do effect sizes in different ways in repeated samples, I have taken the mean difference divided by the standard deviation of the differences (I'll call this d, though maybe I should call it something else?) and get 0.70. I also have a very strong correlation between the samples, not sure if that is problematic.

I would like to put confidence limits around my effect size estimate. To do so, I randomly resample from the difference scores, compute d in the same way and repeat 1000 times. My question is whether this is a good approach, rather than, say, just giving confidence limits around the unstandardised difference or resampling from the original samples. My bootstrap gives me a mean d of 0.79 with confidence limits of [0.4, 1.4]. I've tried this on other random data too. Why am I getting a consistently higher d from bootstrapping, and why are the intervals asymmetric? Is this because of skew in the (difference) scores, and does this make this approach more or less robust?

Thanks!