Using bootstrap I calculate p values of significance tests using two methods:
- resampling under the null hypothesis and counting the outcomes at least as extreme as the outcome coming from the original data
- resampling under the alternative hypothesis and counting the outcomes at least as distant from the original outcome as the value corresponding to the null hypothesis
I believe that the 1st approach is entirely correct as it follows the definition of a p value. I'm less sure about the second, but it usually gives very similar results and reminds me a Wald test.
Am I right? Are both methods correct? Are they identical (for large samples)?
Examples for the two methods (edits after DWin's questions and Erik's answer):
Example 1. Let's construct a bootstrap test similar to the two sample T test. Method 1 will resample from one sample (obtained by pooling the original two). Method 2 will resample from both samples independently.
Example 2. Let's construct a bootstrap test of correlation between x₁…xₐ and y₁…yₐ. Method 1 will assume no correlation and resample allowing for (xₑ,yₔ) pairs where e≠ə. Method 2 will compile a bootstrap sample of the original (x,y) pairs.
Example 3. Let's construct a bootstrap test to check if a coin is fair. Method 1 will create random samples setting Pr(head)=Pr(tail)=½. Method 2 will resample the sample of experimental head/tail values and compare the proportions to ½.