My original dataset is small, 15 observations of variables x and y. I have run Spearmann correlation with and without bootstrapping (B=1000).

I have calculated the bias corrected p value as the number of times the bootstrapped test statistic is greater than or equal to the observed test statistic (+1) dividied by the number of bootstraps (+1)

   P = 1+ #(boot test statistics >= orig test statistic)/B +1

   p=(sum(abs(t_boot) >= abs(t_orig)) +1)/(B+1)

If considering a pair of variables (x and y), then the p value from the calculation above is the likelihood of being able to observe a test statistic greater than or equal to the original test statistic, under the null hypothesis ( that there is no correlation between x and y).

Example result using the above approach (estimate, statistic and p value from orig sample)

    Estimate  Statistic   P.value   Boot P value 
      -0.83    302         0.0054      0.29

So from the boot p value of 0.29 here, it can be inferred that 29%, so 290/1000 bootstraps had a test statistic >= observed, under the null hypothesis. Or could be explained as if there is no correlation between x and y, then we would observe a test statistic of 302 or more by chance, 29% of the time. But how does this affect the interpretation of the original observed correlation between x and y, which was significant, its not intuitive to me. Not fishing for statistical significance, but using this example here for my understanding.

I'm aware of another approach from a post on here regarding p values ( where if feasible for Spearman correlation would use the distribution of (Boot coefficient estimate − original cofficient estimate)/bootstrapped se of coefficient estimate) or ((θ^∗−θ^)/σ^∗) as a reference for comparing against the original value of the t statistic . Correct creation of the null distribution for bootstrapped $p$-values

enter image description here

Any help in interpreting the bootstrap p value in the above example would be appreciated.

  • $\begingroup$ There are lots of bootstrap variations, some more accurate than others. But all are approximations, and the bootstrap distribution may do a poor job in reproducing the sampling distribution. But I don’t have much experience with bootstrap p-values, nor a reason to believe that the standard p-value for correlations doesn’t work well enough. $\endgroup$ Oct 28, 2023 at 19:48
  • 1
    $\begingroup$ Why would you need the bootstrap for a Spearman correlation? (apart from maybe wanting to show that there's problems with the bootstrap, I guess) $\endgroup$
    – Glen_b
    Oct 28, 2023 at 22:55
  • $\begingroup$ @Glen_b It would be more for simulating a greater number of samples, but understandably the starting number of only 15 is small to begin with for the resampling with replacement anyway. Wasn't sure if you meant that the issue lies with the choice of the Spearman correlation (not Pearson for instance)? $\endgroup$
    – aim6789
    Oct 29, 2023 at 10:09
  • $\begingroup$ I don't see a single advantage over an exact permutation test. $\endgroup$
    – Michael M
    Oct 29, 2023 at 14:37
  • $\begingroup$ @MichaelM, I understand - looking for resources for permutation tests for correlation in R. From cor.test, you get estimate(correlation coefficient), test statistic and p value. Unsure as to why p value calculation is using the estimate not the test statistic e.g dgarcia-eu.github.io/SocialDataScience/5_SocialNetworkPhenomena/… and if the null hypothesis from cran.r-project.org/web/packages/flipr/vignettes/… states that the two samples come from the same distribution, how does this affect their correlation? $\endgroup$
    – aim6789
    Oct 29, 2023 at 15:14

1 Answer 1


With an overall sample size of 15 there will always be limits to how well you can approximate anything via resampling. That said, the bootstrap is not suited to approximate the P-value for $H_0: \rho=0$, because the bootstrap does not generate data under the null hypothesis. It would be more appropriate to estimate a standard error and/or confidence interval from your bootstrap, but again, $n=15$.

If you want to resample a P-value you'll need a permutation test, which works directly under a null hypothesis of exchangeability. Basically, you resample one of the two variables without replacement and allocate them to the fixed original other variable, so each X receives a new Y in every sample. In this scheme you can count how many resamples have a statistic that's larger than the original sample. See also here for more explanation and example code.

  • $\begingroup$ thank you for this - will look into CI intervals from bootstrap. Seems there are also different ways to calculate p values following permutation. In the examples I've found (e.g. davidzeleny.net/wiki/doku.php/recol:monte-carlo-pearson) they count how many times the absolute permuted coefficients are >= absolute observed coefficients / number of permutations + 1. Please could you let me know which one would be the "correct" one to use in my case? $\endgroup$
    – aim6789
    Oct 28, 2023 at 21:20
  • $\begingroup$ the other question I had is to check the CI from the bootstrap (need to check on which way Basic vs Percentile vs Normal vs Bias-Corrected) -is indicating that there is 95% confidence that the true/population correlation coefficient between x and y is between the (lower CI, upper CI) $\endgroup$
    – aim6789
    Oct 28, 2023 at 21:25
  • 1
    $\begingroup$ On the real line it shouldn't matter whether you test for > or >=, and the impact of +1 in the denominator will go to zero as permutations go to infinity (though you don't have effectively infinite samples), if you want to be conservative you use >= and +1. Bootstrap intervals all have their own shortcomings; basic or bias-corrected (not BCa) are probably least likely to go really pathologic but are also on the conservative side. $\endgroup$
    – PBulls
    Oct 29, 2023 at 6:56
  • $\begingroup$ was wondering why not BCa, based on this link r-bloggers.com/2019/09/…, it advises this one over the others $\endgroup$
    – aim6789
    Oct 29, 2023 at 14:26
  • $\begingroup$ You seem to ignore the excellent answer. Why trying to make the wrong approach (Bootstrap for testing) a bit less wrong when there is a good one (permutation test)? $\endgroup$
    – Michael M
    Oct 29, 2023 at 14:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.