0
$\begingroup$

I need to compare survey responses from my experimental (referred as "FS" below) and control group (referred as "GV" below) using independent samples t-test. As the two groups have unequal sample size (n1=87,n2=64), I'm thinking about using bootstrapping to compensate for it, assuming our sample is representative of the population.

I will list responses of both groups to one survey question (Question 4, referred as Q4 below) as an example of how I did bootstrapping and t-test. All questions are asked to respond on a 7-point Likert scale.

## what I have done for bootstrapping on R
## FS (experimental group) has a total number of 87 responses, listed as below 
Q4_FS <- c(2,3,3,3,3,4,4,4,4,4,4,4,4,4,4,4,4,5,5,5,5,5,5,5,5,5,5,6,6,6,
           6,6,7,7,7,7,3,3,4,4,4,4,6,6,6,7,7,7,2,3,3,3,3,4,4,4,5,5,5,
           5,5,6,6,6,7,7,7,2,2,3,3,3,3,3,3,3,3,4,4,4,5,5,5,6,6,6,6,6,7)

## to resample with replacement by bootstrapping
mQ4FS <- c()
for (i in 1:1000) {
  Q4FS  <- sample(Q4_FS,87,replace=T)
  mQ4FS <- c(mQ4FS,Q4FS)
}

## GV (control group) has a total number of 62 responses, listed as below 
Q4_GV <- c(5,4,5,3,5,5,4,4,5,4,7,5,3,3,4,3,5,4,4,5,4,7,7,7,5,4,7,2,
           3,2,3,2,3,4,5,6,5,3,5,6,4,5,7,4,2,6,7,6,6,4,3,5,5,6,3,4,4,3,6,6,3,3)

## resample with replacement
mQ4GV <- c()
for (i in 1:1000) {
  Q4GV  <- sample(Q4_GV,62,replace=T)
  mQ4GV <- c(mQ4GV,Q4GV)
}

## perform independent samples t-test 
t.test(mQ4GV,mQ4FS)

## results of t-test
Welch Two Sample t-test

data:  mQ4GV and mQ4FS
t = -20.032, **df = 133890**, p-value < 2.2e-16
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -0.1635008 -0.1343572
sample estimates:
mean of x mean of y 
 4.486887  4.635816 

The unusually large degree of freedom indicates that by "resampling", I increased both samples from a size of 62 and 84 to 62000 and 84000. I am wondering how I could avoid this? Is there any other code I can use to bootstrap?

$\endgroup$
  • 1
    $\begingroup$ @Dole IMHO this question has more statistic that programming issues in it, so it is on-topic. $\endgroup$ – Tim Nov 4 '15 at 20:18
  • 2
    $\begingroup$ It's not clear to me why there's any need to "compensate" for a difference in sample size. The t-test already does that. $\endgroup$ – Glen_b Nov 4 '15 at 20:24
  • 1
    $\begingroup$ Although this question asks for code, the OP's difficulties are due to misunderstandings about the nature of bootstrapping. I think this Q will be better addressed here (as a statistical question) than on Stack Overflow (as a coding question). $\endgroup$ – gung Nov 4 '15 at 20:41
  • $\begingroup$ Thanks @gung. could you elaborate on "misunderstandings about the nature of bootstrapping"? $\endgroup$ – Ariana K. Nov 4 '15 at 21:27
1
$\begingroup$

Bootstrapping like that is giving the t-test the impression that there is more data than there actually is. With lots of data, even small differences are seemingly significant. You should try just running the t test:

Q4_FS <- c(2,3,3,3,3,4,4,4,4,4,4,4,4,4,4,4,4,5,5,5,5,5,5,5,5,5,5,6,6,6,
       6,6,7,7,7,7,3,3,4,4,4,4,6,6,6,7,7,7,2,3,3,3,3,4,4,4,5,5,5,
       5,5,6,6,6,7,7,7,2,2,3,3,3,3,3,3,3,3,4,4,4,5,5,5,6,6,6,6,6,7)

Q4_GV <- c(5,4,5,3,5,5,4,4,5,4,7,5,3,3,4,3,5,4,4,5,4,7,7,7,5,4,7,2,
       3,2,3,2,3,4,5,6,5,3,5,6,4,5,7,4,2,6,7,6,6,4,3,5,5,6,3,4,4,3,6,6,3,3)


## perform independent samples t-test 
t.test(Q4_GV,Q4_FS)

From Wikipedia:

Welch's t-test is more robust than Student's t-test and maintains type I error rates close to nominal for unequal variances and for unequal sample sizes.

$\endgroup$
  • 1
    $\begingroup$ Thanks Andrew! I am using the same methods to compare the results of a voting activity. I just posted this question on stackexchange. Could you please share with me your thoughts? Here is the link: stats.stackexchange.com/questions/180258/… TONS OF THANKS! $\endgroup$ – Ariana K. Nov 5 '15 at 1:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.