# Understanding p value from bootstrapping correlations between pairs of variables

My original dataset is small, 15 observations of variables x and y. I have run Spearmann correlation with and without bootstrapping (B=1000).

I have calculated the bias corrected p value as the number of times the bootstrapped test statistic is greater than or equal to the observed test statistic (+1) dividied by the number of bootstraps (+1)

   P = 1+ #(boot test statistics >= orig test statistic)/B +1

p=(sum(abs(t_boot) >= abs(t_orig)) +1)/(B+1)


If considering a pair of variables (x and y), then the p value from the calculation above is the likelihood of being able to observe a test statistic greater than or equal to the original test statistic, under the null hypothesis ( that there is no correlation between x and y).

Example result using the above approach (estimate, statistic and p value from orig sample)

    Estimate  Statistic   P.value   Boot P value
-0.83    302         0.0054      0.29


So from the boot p value of 0.29 here, it can be inferred that 29%, so 290/1000 bootstraps had a test statistic >= observed, under the null hypothesis. Or could be explained as if there is no correlation between x and y, then we would observe a test statistic of 302 or more by chance, 29% of the time. But how does this affect the interpretation of the original observed correlation between x and y, which was significant, its not intuitive to me. Not fishing for statistical significance, but using this example here for my understanding.

I'm aware of another approach from a post on here regarding p values ( where if feasible for Spearman correlation would use the distribution of (Boot coefficient estimate − original cofficient estimate)/bootstrapped se of coefficient estimate) or ((θ^∗−θ^)/σ^∗) as a reference for comparing against the original value of the t statistic . Correct creation of the null distribution for bootstrapped $p$-values

Any help in interpreting the bootstrap p value in the above example would be appreciated.

• There are lots of bootstrap variations, some more accurate than others. But all are approximations, and the bootstrap distribution may do a poor job in reproducing the sampling distribution. But I don’t have much experience with bootstrap p-values, nor a reason to believe that the standard p-value for correlations doesn’t work well enough. Commented Oct 28, 2023 at 19:48
• Why would you need the bootstrap for a Spearman correlation? (apart from maybe wanting to show that there's problems with the bootstrap, I guess) Commented Oct 28, 2023 at 22:55
• @Glen_b It would be more for simulating a greater number of samples, but understandably the starting number of only 15 is small to begin with for the resampling with replacement anyway. Wasn't sure if you meant that the issue lies with the choice of the Spearman correlation (not Pearson for instance)? Commented Oct 29, 2023 at 10:09
• I don't see a single advantage over an exact permutation test. Commented Oct 29, 2023 at 14:37
• @MichaelM, I understand - looking for resources for permutation tests for correlation in R. From cor.test, you get estimate(correlation coefficient), test statistic and p value. Unsure as to why p value calculation is using the estimate not the test statistic e.g dgarcia-eu.github.io/SocialDataScience/5_SocialNetworkPhenomena/… and if the null hypothesis from cran.r-project.org/web/packages/flipr/vignettes/… states that the two samples come from the same distribution, how does this affect their correlation? Commented Oct 29, 2023 at 15:14

With an overall sample size of 15 there will always be limits to how well you can approximate anything via resampling. That said, the bootstrap is not suited to approximate the P-value for $$H_0: \rho=0$$, because the bootstrap does not generate data under the null hypothesis. It would be more appropriate to estimate a standard error and/or confidence interval from your bootstrap, but again, $$n=15$$.