# Strange behavior of location-tests for different datasets

I have two datasets, $D1$ and $D2$, each containing many samples $S_i$ of 100 numbers, each sample from a different (unknown) population. For example:

$S_1=\{0.125, 0.122, 0.053, \dots\} \\ S_2=\{0.001, 0.013, 0.034, \dots\} \\ \dots$

The are both quite large. $D1$ contains almost 6M samples, and $D2$ contains almost 500K.

I randomly split each sample $S_i$ in two disjoint subsamples of size 50 each, let them be $S_i'$ and $S_i''$. For each subsample I run a t-test for the null hypothesis of mean equal zero, so I have two p-values: $p_i'$ and $p_i''$.

My question of interest is: how often do I observe significance in $S_i''$ if I observed significance in $S_i'$. So, for different significance levels, I plot this proportion:

$\displaystyle\frac {\#\{p_i' \leq \alpha~and~p_i''\leq\alpha\}} {\#\{p_i' \leq \alpha\}}$

With dataset $D1$ there is nothing rare: the lower the significance level the smaller the proportion because it is harder to observe significance. But when I plot it with dataset $D2$, I have this weird behavior in the plot: it kind of goes up with alpha, but then at some point it goes down with large alphas.

I can't figure out why this is happening. The same happens with other tests such as Wilcoxon and Sign. Any ideas?

• I'm afraid I can't follow what you're doing. Perhaps you could illustrate with a tiny version of your data, showing us more clearly what you mean by a "dataset," what you mean by a "sample," exactly how you perform the "splitting," how you manage to obtain two p-values for each t-test, and what distinction (if any) you are making between "p-value," "significance," "significants," and "alpha."
– whuber
Commented Jul 4, 2013 at 16:48
• @whuber Edited the question Commented Jul 4, 2013 at 17:02
• Do both data set have the same number of samples?
– Gala
Commented Jul 4, 2013 at 17:21
• @GaëlLaurans No, but they're both quite large: D1 has almost 6M samples and D2 has almost 500K Commented Jul 4, 2013 at 17:26
• Is your splitting procedure randomized or not?
– whuber
Commented Jul 4, 2013 at 20:16