I ran a simple randomized experiment with 1 control and 1 experimental condition (final N = 80). The dependent variable (frequency of a shown behavior) is clearly not normally distributed so I thought about bootstrapping my analysis (an independent t-test).

The t-test without bootstrapping resulted in a significant effect between the conditions (p < .05). Using SPSS, the p-value based on bootstrapping (5000 resamples) is only marginally significant (p < .10). However, the 95% confidence interval does not include zero.

I just apply statistical methods and sometimes I don't know what is actually right (unfortunately!). But that's why I ask this question. When I tried to learn how Bootstrapping works, I thought that one has to look at the confidence intervals to detect whether an effect is not zero. In my example above, the 95% CI does not accompany with the bootstrapped p-value. So I don't know whether I should report the bootstrapped CI and/or the bootstrapped p-value and/or the typical unbootstrapped p-value.

What would you say?

  • $\begingroup$ I am not an SPSS expert, but to me it looks like it will be difficult to help you if people do not know what is your data, how you bootstrapped it and what score you measured in the end. $\endgroup$
    – gui11aume
    May 31, 2012 at 9:46

2 Answers 2


You have many choices for bootstrap confidence intervals. All bootstrap confidence intervals are approximate and do not always do well in small samples (usually 80 is not considered small). also if you read Hall and Wilson's paper you will find that testing hypotheses assuming the bootstrap distribution under the null hypothesis works better than inverting confidence intervals. It is an issue about how to center the pivotal quantity in the test statistics. Schenker in 1985 showed that bootstrap methods such as Efron's percentile method and even the BC method severely under cover the true parameter for certain chi square populations when the sample size is not very large. Chernick and LaBudde in 2010 American Journal of Mathematical and Management Science showed that in small samples there can even be problems with BCa and bootstrap t for highly skewed distributions such as the lognormal. So based on the literature including my own research I suggest doing the hypothesis test with the centering approach recommended by Hall and Wilson and base your conclusions on that p-value. You can find detailed coverage of this in my recent book "An Introduction to the Bootstrap with Applications to R" published by Wiley in 2011.

  • $\begingroup$ Since this isn't an academic paper, a link to referenced articles is better than a citation IMO. $\endgroup$ May 31, 2012 at 15:29
  • 3
    $\begingroup$ Yes but it also takes very little effort to use the citation on Google to find any link I might give and others. Some of the links might have the paper available to read either for free or for some small fee. $\endgroup$ May 31, 2012 at 15:41
  • $\begingroup$ If the reference is to "Two Guidelines for Bootstrap Hypothesis Testing" by Peter Hall and Susan R. Wilson, Biometrics Vol. 47, No. 2 (Jun., 1991), pp. 757-762, then this paper is available on JSTOR. They charge for download, but the paper can be read for free on the website. $\endgroup$
    – A. Donda
    Nov 7, 2014 at 16:40
  • $\begingroup$ @MichaelChernick I have just started a bounty here: stats.stackexchange.com/questions/175659/… after going through chapter 3 of your book - I am unclear under what conditions can bootstrap CIs be used for hypothesis testing ? Does the statistic need to be pivotal ? Does the distribution need to be symmetrical ? Thanks ! $\endgroup$ Jun 14, 2019 at 16:13
  • 1
    $\begingroup$ There is a 1-1 correspondence in general between confidence intervals and hypothesis tests. For example a 95% confidence interval for a model parameter represents the non-rejection region for the corresponding 5% level hypothesis test regarding the value of that parameter. There is no requirement about the shape of the population distributions. Obviously if it applies to confidence intervals in general it will apply to bootstrap confidence intervals. $\endgroup$ Jun 14, 2019 at 17:27

I am not a real Bootstrap expert, but I can tell you about the two main things:

  1. Bootstrap confidence intervals are usually more robust and accurate then the ones estimated without bootstrap.
  2. If you estimate the parameter with bootstrap, your confidence interval (CI) usually evaluated in a different way then in a regular t-test. For example, in a regular case CI is $[ \hat{\theta} - \hat{q}_{1-\alpha/2}, \hat{\theta} + \hat{q}_{\alpha/2} ]$ (here $\hat{\theta}$ is an estimate of the parameter, $\hat{q}_{\alpha}$ is an $\alpha$-quantile). But for bootstrap it is $[ \hat{\theta} - \hat{q}_{1-\alpha/2}, \hat{\theta} - \hat{q}_{\alpha/2} ]$ (minus sign in both cases).

From that all I would suggest you to recheck whether with this formulas bootstrap CI accompany with the p-value. And if you will find that it's ok now, report bootstrapped results. If no, it is better to ask SPSS experts about how bootstrap works there.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.