Consider two samples ${(x_i)}_{i=1}^M$ and ${(y_i)}_{i=1}^N$, where $M = N$, respectively of two sequence of independent but not identically distributed random variables $(X_i)_{i=1}^M$ and $(Y_i)_{i=1}^M$, but the samples are paired in the sense that $X_i$ is not independent from $Y_i$. I would like to know whether the sample means $\bar{x}$ and $\bar{y}$ are significantly different from each other, by using bootstrapping. I will describe two plausible methods to do this. I am confused as to which method is more appropriate. This question is related to the following questions, however in these cases $M \neq N$:

Non-parametric confidence interval about the difference of means for unpaired data

Confidence interval for the difference of two means using boot package in R

Method 1: This method treats the data as unpaired as per the above questions.

  1. Sample $M$ points with replacement from ${(x_i)}_{i=1}^M$, calculate the bootstrapped mean $\bar{x}^*$.
  2. Sample $M$ points with replacement from ${(y_i)}_{i=1}^M$, calculate the bootstrapped mean $\bar{y}^*$.
  3. Calculate the bootstrap difference in means $\delta^* = \bar{x}^* - \bar{y}^*$.
  4. Repeat steps 1-3 as many times as necessary to get the bootstrap population of the bootstrap difference, and construct the 95% confidence interval as usual.

Method 2: This method treats the data as paired.

  1. Sample $M$ indices with replacement from the indices $1, 2, \dots, M$.
  2. Construct samples from ${(x_i)}_{i=1}^M$ and ${(y_i)}_{i=1}^M$ by taking the elements of the sequence given by the indices from step 1. Find the bootstrap means $\bar{x}^*$ and $\bar{y}^*$ from these samples.
  3. Repeat steps 1-2 as necessary, and construct the 95% confidence interval as usual.

The difference between Method 1 and 2 is that in method one the two bootstrap samples are derived independently.

Example use case:

We have two statistical models $\mathcal{P_1}=\{P_{\theta_1}\}$ and $\mathcal{P_2}=\{P_{\theta_2}\}$ respectively parametrized by $\theta_1$ and $\theta_2$ for a sequence of independent, but not identically distributed, random variables $(Z_i)_{i=1}^M$. For a sequence of sammples $(z_i)_i$ of $(Z_i)_i$, we can calculate the log-likelihood of each model, for each observation $z_i$. These log-likelihoods are are samples of the random variable representing the log-likelihood of each model. Then, I would like to see whether the mean log-likelihood value calculated from these samples significantly differ from each other.

  • $\begingroup$ Hi, I have made an edit to describe how the data is paired. $\endgroup$ – Alex Sep 7 '16 at 0:03
  • 3
    $\begingroup$ Are you effectively asking "If my data are paired is it okay to ignore that pairing when trying to calculate an interval for the difference in means?" $\endgroup$ – Glen_b Sep 7 '16 at 0:07
  • 1
    $\begingroup$ Thanks for the edit. For paired data you should use Method #2. You can think about it like that: if the data are paired, you can compute differences $x_i - y_i$ and then bootstrap those to get a CI of the differences distribution. Your Method #2 is equivalent to that. $\endgroup$ – amoeba Sep 7 '16 at 0:07
  • $\begingroup$ @Glen_b Yes that is a good way of putting it. $\endgroup$ – Alex Sep 7 '16 at 0:08
  • $\begingroup$ Ah, it looks the question is answered if you consider it as a generalisation of the difference between paired and unpaired t-tests: stats.stackexchange.com/questions/38102/… $\endgroup$ – Alex Sep 7 '16 at 0:25

The first method is no resampling test of which I'm aware in the literature. It seems like your goal, by resampling $X$ and $Y$ independently, is to generate data under the null hypothesis. This approach is inefficient because you are ignoring pairing in the design.

The preferred resampling method for generating data under the null hypothesis is the permutation test. Permutation testing for paired data is done by randomly negating the $X-Y$ differences; i.e. replacing them with $Y-X$. Here, the between-pair differences are preserved, but the within-pair differences are only preserved if the paired mean difference is 0.

The second example is a proper description of a paired bootstrap.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.