# Compute-intensive randomized *unpaired* t-test (without paired samples)

Yeh 2000 section 3.3 cites Noreen (1989) and Cohen (1995) for compute-intensive, versions of paired t-tests, which are stonger because they don't make parametric assumptions. However, these tests require paired samples:

"When comparing two techniques, we gather-up all the responses (whether actually of interest or not) produced by one of the two techniques when examining the test data, but not both techniques. Under the null hypothesis, the two techniques are not really different, so any response produced by one of the techniques could have just as likely come from the other. So we shuffle these responses, reassign each response to one of the two techniques (equally likely to either technique) and see how likely such a shuffle produces a difference (new technique minus old technique) in the metric(s) of interest (in our case, precision and F-score) that is at least as large as the difference observed when using the two techniques on the test data."

How can I apply a similar compute-intensive test when computing a set of responses, when there is no pairing between different sets of responses? i.e. what is the compute-intensive randomized version of an unpaired t-test?

For example, I have set of responses A and set of responses B. I would like to use a compute-intensive randomized test to see if the mean of A is statistically significantly higher than the mean of B?

Is there an analogous compute-intensive statistical test I can use, that avoids the paired-example-assumption intrinsic in stratified shuffling?

Suppose we have the two sample x1 and x2 of sizes $$n_1=20, n_2 = 25$$ with summaries and stripcharts as shown below:

summary(x1)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
21.83   36.33   51.70   50.83   61.63  102.32
summary(x2)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
11.28   27.82   36.17   37.55   44.18   78.84

stripchart(list(x1,x2), ylim=c(.5,2.5), pch="|")


We want to test $$H_0: \mu_1=\mu_2$$ against $$H_a: \mu_1\ne\mu_2.$$ The data seem sufficiently right skewed that I would feel uncomfortable using a 2-sample t test. However, we note that both the Welch (default) and pooled versions (parameter var.eq = t) of the test show P-values below 0.05, as shown below. The difficulty is that I wonder whether the t statistics are t distributed, and so I wonder if I can trust these P-values.

I have no problem with the pooled t statistic as a way of measuring the distance between the two sample means, I wonder only about the distribution of the "t statistic." So I will use the pooled t statistic as my 'metric' for a permutation test.

A permutation test would randomly assign 20 of the 45 observations to group 1 and the rest to group 2, and find the P-value of (say) the pooled test. By repeatedly scrambling the 45 observations among the two groups and finding the resulting P-value after each permutation, I could get a good idea of the distribution of P-values under $$H_0.$$ Then I can see where the P-value for the actual data stands in the simulated permutation distribution. [By intricate combinatorics I might be able to derive the exact permutation distribution of the t statistic under the $${45\choose 20}$$ permutations of the data, but with a few thousand permutations I can get reasonably close approximation.]

x = c(x1, x2)
g = rep(1:2, c(20,25))
pv.obs = t.test(x ~ g)$p.val; pv.obs [1] 0.02379766 set.seed(2021) pv.prm = replicate(5000, t.test(x~sample(g))$p.val)
mean(pv.prm < pv.obs)
[1] 0.0232     # simulated P-value of permutation test


Thus, I can feel confident that the sample means are significantly different at the 5% level.

hist(pv.prm, prob=T, br=20, col="skyblue2")
abline(v=pv.obs, col="orange", lwd=3, lty="dotted")


t.test(x1,x2)$$p.val [1] 0.02379766 t.test(x1,x2, var.eq=T)$$p.val
[1] 0.01912915


Notes:

(1) Without additional arguments, the R procedure sample randomly permutes elements of its first argument. Example:

sample(c(1,1,1,2,2,2))
[1] 1 2 2 2 1 1
sample(c(1,1,1,2,2,2))
[1] 2 1 1 2 1 2
sample(c(1,1,1,2,2,2))
[1] 2 2 2 1 1 1
sample(c(1,1,1,2,2,2))
[1] 2 1 1 2 2 1


(2) Here is R code used to make the two hypothetical samples:

set.seed(409)
x1 = rgamma(20, 5, .11)
x2 = rgamma(25, 5, .12)

• A followup question, if you don't mind. I have 6 corpora of 2000 samples. For each corpora pair, I select 1000 samples for each corpus and compute the Maximum Mean Discrepancy. I do 300 trials per corpus pair to get the mean MMD for a corpus pair. Does it make sense to do this sort of permutation test, scrambling the 300 corpus pair MMD means to see if the MMD discrepancy between two corpora are statistically significant? Commented Apr 10, 2021 at 15:08