p-value vs. confidence interval obtained in Bootstrapping

Question

I ran a simple randomized experiment with 1 control and 1 experimental condition (final N = 80). The dependent variable (frequency of a shown behavior) is clearly not normally distributed so I thought about bootstrapping my analysis (an independent t-test).

The t-test without bootstrapping resulted in a significant effect between the conditions (p < .05). Using SPSS, the p-value based on bootstrapping (5000 resamples) is only marginally significant (p < .10). However, the 95% confidence interval does not include zero.

I just apply statistical methods and sometimes I don't know what is actually right (unfortunately!). But that's why I ask this question. When I tried to learn how Bootstrapping works, I thought that one has to look at the confidence intervals to detect whether an effect is not zero. In my example above, the 95% CI does not accompany with the bootstrapped p-value. So I don't know whether I should report the bootstrapped CI and/or the bootstrapped p-value and/or the typical unbootstrapped p-value.

What would you say?

Answer 1

You have many choices for bootstrap confidence intervals. All bootstrap confidence intervals are approximate and do not always do well in small samples (usually 80 is not considered small). also if you read Hall and Wilson's paper you will find that testing hypotheses assuming the bootstrap distribution under the null hypothesis works better than inverting confidence intervals. It is an issue about how to center the pivotal quantity in the test statistics. Schenker in 1985 showed that bootstrap methods such as Efron's percentile method and even the BC method severely under cover the true parameter for certain chi square populations when the sample size is not very large. Chernick and LaBudde in 2010 American Journal of Mathematical and Management Science showed that in small samples there can even be problems with BCa and bootstrap t for highly skewed distributions such as the lognormal. So based on the literature including my own research I suggest doing the hypothesis test with the centering approach recommended by Hall and Wilson and base your conclusions on that p-value. You can find detailed coverage of this in my recent book "An Introduction to the Bootstrap with Applications to R" published by Wiley in 2011.

Answer 2

I am not a real Bootstrap expert, but I can tell you about the two main things:

1. Bootstrap confidence intervals are usually more robust and accurate then the ones estimated without bootstrap.
2. If you estimate the parameter with bootstrap, your confidence interval (CI) usually evaluated in a different way then in a regular t-test. For example, in a regular case CI is $[ \hat{\theta} - \hat{q}_{1-\alpha/2}, \hat{\theta} + \hat{q}_{\alpha/2} ]$ (here $\hat{\theta}$ is an estimate of the parameter, $\hat{q}_{\alpha}$ is an $\alpha$-quantile). But for bootstrap it is $[ \hat{\theta} - \hat{q}_{1-\alpha/2}, \hat{\theta} - \hat{q}_{\alpha/2} ]$ (minus sign in both cases).

From that all I would suggest you to recheck whether with this formulas bootstrap CI accompany with the p-value. And if you will find that it's ok now, report bootstrapped results. If no, it is better to ask SPSS experts about how bootstrap works there.