This is a question regarding the t-test in SPSS.

I have two groups and I want to test if the two means are equal. I am using the t-test with bootstrapping. In the end I got a p-value<0.005, which would generally cause me to reject the null hypothesis that the means of the two populations are equal but in my case the zero lies within the 95% BCa bootstrap confidence intervals based on 1000 samples.

Do I still reject the hypothesis of equal means?

  • 1
    $\begingroup$ To clarify, did you conduct a bootstrapped t-test from which you are now comparing the p value and 95%CI, or did you run a standard t-test (not bootstrapped) to get the p-value and only used bootstrapping for the CI? $\endgroup$ – Rose Hartman Sep 14 '17 at 8:03

Caveat: This answer assumes that the question is about interpreting bootstrapped p-values and CIs. A comparison between a traditional p-value (not bootstrapped) and a bootstrapped CI would be a different issue.

With a traditional (not bootstrapped) t-test, the 95%CI and the p-value's position relative to the .05 cutoff for significance will always tell you the same thing. That's because they're both based on the same information: the t-distribution for your degrees of freedom and the mean and standard error observed in your sample (or difference between means and standard error, in the case of a two-sample t-test). If your CI doesn't overlap with 0, then your p-value will necessarily be < .05 --- unless, of course, there's a bug in the software or a user error in implementation or interpretation of the test.

With a bootstrapped t-test, the CI and p value are both calculated directly from the empirical distribution generated by the bootstrapping: the p value is simply what percent of bootstrapped group differences are more extreme than the original observed difference; the 95%CI is the middle 95% of bootstrapped group differences. It is not impossible for the p-value and the CI to disagree about significance in a bootstrapped test.

Do you accept or reject the null hypothesis?

In the context of a bootstrapped test, the p-value (as compared to the CI) more directly reflects the spirit of the hypothesis test, so it makes the most sense to rely on that value to decide whether or not to reject the null at your desired alpha (generally .05). So in your case, where the p value is less than .05 but the 95%CI contains zero, I recommend rejecting the null hypothesis.

All of this skips over the big ideas about how important "significance" really should be and whether or not null hypothesis significance testing is actually that useful of a tool. Briefly, I always recommend complimenting any significance testing analysis with estimation of effect sizes (for a two-sample t-test, the best effect size estimate will probably be Cohen's d), which can provide some additional context to help you understand your results.

Related helpful post: What is the meaning of a confidence interval taken from bootstrapped resamples?

| cite | improve this answer | |
  • 1
    $\begingroup$ This is a great answer (+1), but some advice on how the OP approaches whether he accepts or rejects the Null would round out the answer for the OP's final question. $\endgroup$ – Ashe Sep 13 '17 at 13:21
  • $\begingroup$ @Ashe Thanks! You're right that I didn't address the central question head-on. I'll edit to improve that. $\endgroup$ – Rose Hartman Sep 13 '17 at 13:24
  • $\begingroup$ "for a two-sample t-test, the best effect size estimate will probably be Cohen's d" Is this specific to the bootstrapping? Because I would think for a normal t-test, the confidence interval is gives you the best information about effect size on the actual scale that you tested on. $\endgroup$ – David Ernst Sep 13 '17 at 14:08
  • $\begingroup$ Cohen's d is for any two group difference; bootstrapping or not is irrelevant. CIs are generally not considered estimates of "effect size" since they depend on sample size (e.g. en.wikipedia.org/wiki/…: "Unlike the t-test statistic, the effect size aims to estimate a population parameter and is not affected by the sample size.") Perhaps what you're wondering about is standardized vs. unstandardized effect size estimates? The unstandardized effect size for two groups is just the raw difference between means. $\endgroup$ – Rose Hartman Sep 13 '17 at 14:16
  • $\begingroup$ Many thanks! Your explanation about what p-value and CI are in the context of a bootstrapped t-test was very useful. As you suggest I determined the Cohen's d, a very helpful statistic in understanding my results. $\endgroup$ – Liza Vieira Sep 13 '17 at 14:41

If the p-value of the null hypothesis is smaller than 0.05, then zero should not be contained in the confidence interval at 0.05 of the parameter that you are assuming to be zero in the null hypothesis. This is the same thing. So there is a bug or you don't test the same hypothesis.

EDIT, as the other answers and the comment below correctly indicate, this is not the full story. However, I still think that if one test indicates groups have different mean (p < 0.005), and the other does not reject (p > 0.05), probably the tests are really checking a different thing.

While theoretically this difference could be due to asymptotics (bootstraps are approximations on finite sample, other tests are approximations based on normality assumptions), that difference is surprisingly large. I argue it is alarmingly large, and without figuring out what is going on with that, you should not yet draw conclusions. That is also exactly what you are doing, by the way, by posting the question here. Maybe you can share the numbers and make this interesting question a bit more concrete.

| cite | improve this answer | |
  • 2
    $\begingroup$ I disagree. A bootstrapped confidence interval may not follow the results of a t-test, as it is a different kind of procedure altogether (in this case based on the difference of group means). Especially when 'bias-corrected and accelerated bootstrap confidence interval is made, things like asymmetric confidence intervals around the original estimate (i.e. difference of group means in this case) could occur. $\endgroup$ – IWS Sep 13 '17 at 13:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.