I'm having trouble understanding the difference between two bootstrap procedures to evaluate the difference in means between two samples. As an example, consider the following scenario:

My goal is to compare the happiness between two countries, Nepal and Bhutan. I've collected a sample of $N$ happiness ratings from Bhutan and $N$ happiness ratings from Nepal. Each data point is IID.

Procedure 1: Bootstrap while Retaining Country Membership

Following this guide (page 4-5), the procedure is:

  1. Fit a regression model $Y = \beta_0 + \beta_1X_1$ using the observed samples to estimate the observed difference $\beta_1$
  2. Pool citizens from Nepal and Bhutan into an overall population, maintaining their country membership
  3. Draw with replacement $2N$ samples from the combined population and fit a regression model to this bootstrapped sample
  4. Repeat step 2 $B$ times to generate a distribution of $\beta_1$ values
  5. Use the bootstrapped distribution to compute p-values and confidence intervals for $\beta_1$

Procedure 2: Draw From Overall Sample without Country Membership

Following these lecture notes (slides 28-32) from a prior probability class I took, the procedure is:

  1. Compute the observed difference in means
  2. Pool all citizens from both countries into a single population, ignoring country membership
  3. Draw with replacement $N_{Nepal}$ citizens from the combined population for the bootstrapped Nepal group
  4. Draw with replacement $N_{Bhutan}$ citizens from the combined population for the bootstrapped Bhutan group
  5. Compute the difference in means between the bootstrapped Nepal and Bhutan groups
  6. Repeat steps 3-5 $B$ times
  7. Compute p-values and confidence intervals using the bootstrapped distribution of differences

Country membership may be changed in procedure 2 so that some happiness ratings from Bhutan appear to be from Nepal in the bootstrap sample. This never happens in Procedure 1, where all the happiness ratings from Nepal are only ever attributed to Nepal.

What are the implications of "swapping" country membership in Procedure 2? Do confidence intervals and p-values computed from the two procedures represent the same quantities? Should I expect the same results from both procedures?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.