Jackknifing for assessing the "robustness" of test results

Question

In a presentation I saw recently, a two-sided t-test was repeated with jackknifed subsets of the original data in order to assess the result's "robustness".

In detail, they took a random half of the first group and a random half of the second group (with a couple of hundred samples each), ran a t-test, recorded the result, and repeated this for 100 times. Then, they used the count of significant p-values to estimate the robustness of their original result (a t-test with the full dataset).

I am more of a "stats-consumer" with a weak theoretical background and have never seen anything like this before. My impression about jackknifing / bootstrapping was always that you use it to "recreate" a null distribution in order to assess the extremeness of your result, which is quite different to the procedure described above.

However, it seems to be a simple and interesting approach to see whether only a few samples determine the result. Thus, I wanted to read up on the theory behind it (how many samples one should use, what contingencies to consider, etc.) but couldn't really find anything.

Now I am sceptical. Is this actually a valid strategy and if so, does it have a specific name or term associated with it? Could you point me to related literature?

Answer 1

To start, a couple of definitions:

• Full sample: N observations
• Bootstrap sample: m samples of size n from the full sample taken with replacement
• Jackknife sample: N samples of size N-1 where each observation is left out once.

Although the question does not say, I assume that these were bootstrap samples taken with replacement. "In detail, they took a random half of the first group and a random half of the second group (with a couple hundred samples each), ran a t-test, recorded the result, and repeated this for 100 times."

I think that what was described was a method to determine if the p-value of the t-test is robust to the underlying distribution of the data. If the data is normally distributed, and if there is no difference in the distribution between groups, then if you take bootstrap samples from both groups and perform a t-test each time at alpha = 0.05, then you should expect 5 out of 100 bootstrap t-tests to reject the null hypothesis. If the data is highly non-normal, then this procedure will reject more or less than 5% of the time expected. If, however, the two groups are not from the same distribution, this procedure doesn't tell you anything about the robustness of the t-test.