P-values of A/A Test with dependence does not obey uniform distribution? Suppose we have a large population (U.S. people). I divide population into 100 subgroups. Note that all people in each of the subgroups are i.i.d..
Suppose I make 50 tests for mean height between 100 subgroups(Group1 vs. Group2, Group3 vs. Group4, etc.), then I will get 50 independent p-values. When the null hypothesis is true (it really is), p-values obey a 0-1 Uniform distribution.
My question is:

If we make tests between the combination of 100 subgroups, which means 1000*99/2 = 4950 tests(Group1 vs. 2/3/.../100, Group2 vs. 3/4/.../100, etc.). P-values are dependent, and what is the distribution of these dependent A/A Tests' p-values?

From my simulations:

*

*independent p-values obey uniform distribution

*dependent p-values are not(can not pass ks-test)


But I can not prove it, or make a very intuitive explanation for this problem, so I need your help ;)
Code on colab
import scipy.stats as stats
from scipy.stats import ttest_ind
import matplotlib.pyplot as plt
from scipy.stats import kstest,ks_2samp, anderson_ksamp, anderson
from itertools import permutations, combinations

n_bkt = 1000
data = stats.norm.rvs(size =100000)
bucket = np.random.randint(0,n_bkt, size = 100000)
res = []
for i in range(0,n_bkt//2):
    control = data[bucket == (i*2)]
    treatment = data[bucket == (i*2 + 1)]
    res.append(ttest_ind(control, treatment))
    
res2 = []
for i,j in combinations(range(0,n_bkt),2):
    control = data[bucket == i]
    treatment = data[bucket == j]
    res2.append(ttest_ind(control, treatment))
    
    
fig, axs = plt.subplots(2,2,figsize=(10, 8))
fig.suptitle('Two A/A Test methods')

axs[0,0].hist([i[0] for i in res])
axs[0,0].set_title('T-Statistics of Independent A/A Tests')
axs[0,1].hist([i[1] for i in res])
axs[0,1].set_title('P-values of Independent A/A Tests')
axs[1,0].hist([i[0] for i in res2])
axs[1,0].set_title('T-Statistics of Dependent A/A Tests')
axs[1,1].hist([i[1] for i in res2])
axs[1,1].set_title('P-values of Dependent A/A Tests')


# Test whether T-statistic obey standard normal distribution
## independent
print(kstest([i[0] for i in res],lambda x: stats.norm.cdf(x,loc = 0, scale = 1)))
## dependent
print(kstest([i[0] for i in res2],lambda x: stats.norm.cdf(x,loc = 0, scale = 1)))

# Test whether P-values obey 0-1 Uniform Distribution
## independent
print(kstest([i[1] for i in res], lambda x: stats.uniform.cdf(x,loc = 0, scale = 1)))
## dependent
print(kstest([i[1] for i in res2], lambda x: stats.uniform.cdf(x,loc = 0, scale = 1)))

 A: *

*In fact your p-values look at least as uniform for the dependent tests; it seems that K-S rejects uniformity there because of the large sample size (number of p-values), i.e., even a small deviation from uniformity is detected.


*The distribution theory of the t-test requires observations to be distributed Gaussian. As your data are drawn from real observations, they are not precisely Gaussian distributed, and the t-test theory will only hold approximately, resulting in small deviations from the uniform distribution for p-values (regardless of whether tests are dependent or independent, however these deviations may be so small that you didn't detect them with the smaller number of independent tests).


*There is an additional reason why dependent p-values of this kind may be less uniform. If one of your groups has an unusually high "mean height", not only is it involved in 99 tests, also as the mean of all group means is the overall mean, this means that there also must be one or more other groups with exceptionally small means, leading to a somewhat higher amount of p-values near 0 or 1 than expected. On the other hand your 100 groups may be unusually similar regarding their means (if, say, 50 of them are similar to the overall mean, the mean of the remainder also must be similar to the overall mean), in which case fewer p-values close to 0 or 1 may occur than expected. These should be small effects not affecting uniformity big time, but they can happen either way with a larger probability than expected under independence.
