I have two independent samples of observations. From each sample I produce a statistic. Let's denote these as $\theta_1$ and $\theta_2$. I'd like to test the hypothesis that $H_0: \Theta_1=\Theta_2$, but I have these two constraints:

  • There is no analytical estimate of the distributions of $\theta_1$ and $\theta_2$ (the statistic is a product of some (computationally expensive) algorithm that operates on each sample.
  • Even under $H_0$, exchanging observations between the two samples in not sensible. Therefore, a permutation approach might reject $H_0$ erroneously.

My current idea was to bootstrap $\theta_1$ and $\theta_2$ independently and then estimate the distribution of $\Theta_2-\Theta_1$ from these two bootstrapped distributions by means of convolution.

Q1: Is this a valid approach?

Q2: Any reason why not the extend this to jackknifing (instead of bootstrapping) as well?

Q3: Any references to such 'two-samples' bootstrap?

Q4: Any recommended alternatives?

  • $\begingroup$ I'm not sure why you need convolution. You could just bootstrap each simultaneously (if the sample sizes are big enough to support it) and then compute a direct bootstrap distribution of the difference. $\endgroup$ – Glen_b Mar 4 '14 at 22:10
  • $\begingroup$ @Glen_b: By bootstrapping "simultaneously" do you mean to take the differences $\theta_1 - \theta_2$ on each bootstrap iteration? Which is a bit like a paired test, even though the samples are not paired? $\endgroup$ – amoeba Mar 5 '14 at 0:13
  • 2
    $\begingroup$ I mean take the differences $\hat\theta_1-\hat\theta_2$. Within a bootstrap iterate, the computation is actually like an unpaired test, not a paired test ... indeed it's exactly analogous to the numerator of a two-sample t-statistic (where in that case, $\hat \theta_i$ is a sample mean). ... though if you mean 'paired' in the sense that we're pairing by bootstrap iterate, then yes, that's exactly what I mean, but it's not really pairing in the usual sense, since the two are quite independent. It's not necessary to do it this way, but it's certainly convenient. $\endgroup$ – Glen_b Mar 5 '14 at 0:23
  • 1
    $\begingroup$ @Glen_b: 1. thanks, I added this info to the question. 2. Since the two samples are independent, one can resample pairs of $\theta_1$ and $\theta_2$ from the two bootstrapped distributions. This way we can have much more resampled pairs of $\theta_1-\theta_2$ than individual bootstrap iterations. I suspect that if the number of this second stage resamplings is really large, the result is the same as a convolution of the two distributions. $\endgroup$ – Trisoloriansunscreen Mar 5 '14 at 8:36
  • 1
    $\begingroup$ resampling the resamples contains no information not in the original sample -- it would surely be advantageous if it's faster to resample the resamples than to resample the original, but otherwise I'm not sure I see the additional information comes from. (You might be able to get some advantage with a smoothed bootstrap perhaps.) $\endgroup$ – Glen_b Mar 5 '14 at 8:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.