Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 4 down vote favorite

I want to verify using a bootstrap approach the hypothesis about a lack of the intervention effect within a group of six patients: pre-(X) post(Y) intervention measurement. My data

ID      X      Y
1  9.856  8.992
2 19.512  4.573
3  1.936  1.572
4 14.575  1.529
5  8.476 12.000
6  1.862  1.417

Within R (2.15.1) Ive written the following code, using the t-test for paired data, which relies on resampled pairs:

boot.p.value <- function(data, S)
{
    boot.t.stat <- as.numeric()
    t.stat <- t.test(x=data[,1], y=data[,2], paired=TRUE)$statistic
        for(s in 1:S)
        {
            boot.data <- data[sample(1:nrow(data), replace=TRUE),] ## resample pairs
            boot.t.stat[s] <- t.test(x=boot.data[,1],y=boot.data[,2], paired=TRUE)$statistic
    }
    p.value <- sum(1*(boot.t.stat >= t.stat))/S
    return(p.value)
}

Where:

boot.p.value(data, S=1000) [1] 0.518

When repeated the resulting p-values values stay between .4 and .6.

For the same data set the SPSS ver. 19 for the paired samples t-test provides bootstrap-based p = 0.182, for 1000 resamples. Why this difference?

share|improve this question
I would like to help but I am confused by your question. I don't understand why you want to bootstrap the t test. I am not able to understand what you are doing just by reading your R code because I am not familiar with what the functions you apply are doing. If a paired t test is appropriate you don't need to do bootstrap. – Michael Chernick Sep 28 '12 at 11:02
1  
If the pairing makes sense and you retain the pairs when you generate bootstrap samples the bootstrap distribution under the null hypothesis that the paired differences have 0 mean should give approximately the same result as the paired t test (but not necessarily if the differences are not normally distributed). The result you get will be a Monte Carlo approximation to the bootstrap and therefore has a small component of error in approximation due to having B (number of bootstrap samples finite). There could many possibilities for differences. – Michael Chernick Sep 28 '12 at 11:07
You have to understand exactly what options you use in each program. One example (a guess to illustrate) would be that one program selects paired data to use for the sampling with replacement while the other separately samples from each group with replacement and then makes the pairs. The first method would be appropriate because it preseves the correlation in the pairs while the second will mess to up (i.e. make a positive correlation nearly 0). This would in essence be like comparing doing a paired t test versus an unpaired one on the original data. – Michael Chernick Sep 28 '12 at 11:14
I fixed up the formatting; since the site automatically puts your name, you don't need it at bottom of post. – Peter Flom Sep 28 '12 at 11:14
1  
It is debatable what is best when the sample size is small. I think all nonparametric tests have little power in small samples. Parametric tests can do a little better if the assumptions are right. But if you want to check assumptions 6 samples is not enough to do that. – Michael Chernick Sep 28 '12 at 13:33
show 6 more comments

1 Answer

up vote 2 down vote accepted

Your bootstrap function is not correct.

I know why all of your p values are between 0.4 and 0.6 and are averaging 0.5: half of your resamples give you a test statistic below and half of your resamples give you a test statistic above the original. You will always get that result from that function - I tried it out with some other data. You aren't randomly switching up the pre and post data.

To get the bootstrap p value, you compare your observed test statistic, 

 t.test(x=data[,1], y=data[,2], paired=TRUE)$statistic

with a random shuffling of pre and post data. So, you need to sample from your original data AND randomly mix up pre and post data (maintaining pairs though).

I'll try to post some code later if you still need help.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.