# How to estimate P-value from bootstrapped distributions?

I have three sets of distributions generated from the bootstrap method.

Normally I would do ANOVA to see if all distribution are equal and then Tukey HSD to get confidence interval and P-value on each pair. However, since the data is generated by bootstrap, I can decide the sample size, in this case it seems my "normal" approach is always significant given the sample size is large enough.

So what is an appropriate method to compare these distributions that are generated by bootstrap?

EDIT: To clarify, by sample size I meant bootstrap resample size.

• I think the standard approach would be to bootstrap the parameter and effect size estimates, so you get parameter estimation confidence intervals. For that, you can for example calculate the point estimates (for parameters and effect sizes) on a few thousand bootstrapped samples and take the 2.5th and 97.5th percentiles of all resulting values. Also, truly random samples drawn from the same population should not result in arbitrarily smaller p values as a function of sample size? You would expect a long-term type I rate of ~5% when drawing two samples from one population and testing at 5%.
– jona
Commented Jul 29, 2013 at 13:45
• Hi Nick! Thank you for the reply and sorry for the confusion. I meant p-value in a t-test gets smaller when I increase the number of times I choose to bootstrap. (Which becomes the sample size when I compare two distribution using t-test that's generated from bootstrapping) Lets say I want to see if the green average is significantly higher than the orange average at 5% alpha in the chart. The density plot is created using 1000 averages from green and 1000 from orange, what additional information do I need? Commented Jul 29, 2013 at 14:09
• 1. as far as I understand it, with every bootstrap sample draw, the lower bound of your p value decreases, but the actual value should asymptotically approach something not zero. 2. AFAIK, standard bootstrap test: calculate mean difference T from full data. Pool green and orange, repeatedly draw two random samples with replacement, calculate the mean difference B, and compare it to full sample mean. For which percentage of your draws was T higher than B? That is your p. 3. Easier to obtain and more informative might be a bootstrapped 95% CI of the mean difference.
– jona
Commented Jul 29, 2013 at 14:46
• Thank you, that made alot of sense. The distribution stays relative same when resample size gets larger so using CI for the difference between two means is quite stable. Now if there is only a way to do calculation similar to Anova and TukeyHSD in R that does the whole group. :) Commented Jul 29, 2013 at 18:55
• Why do you want to bootstrap the ANOVA anyways? Probably, you can just do it analytically, and bootstrap the CI of the effect size.
– jona
Commented Jul 29, 2013 at 20:10