Strange behavior of location-tests for different datasets I have two datasets, $D1$ and $D2$, each containing many samples $S_i$ of 100 numbers, each sample from a different (unknown) population. For example:
$S_1=\{0.125, 0.122, 0.053, \dots\} \\
 S_2=\{0.001, 0.013, 0.034, \dots\} \\
\dots $
The are both quite large. $D1$ contains almost 6M samples, and $D2$ contains almost 500K.
I randomly split each sample $S_i$ in two disjoint subsamples of size 50 each, let them be $S_i'$ and $S_i''$. For each subsample I run a t-test for the null hypothesis of mean equal zero, so I have two p-values: $p_i'$ and $p_i''$.
My question of interest is: how often do I observe significance in $S_i''$ if I observed significance in $S_i'$. So, for different significance levels, I plot this proportion:
$\displaystyle\frac
{\#\{p_i' \leq \alpha~and~p_i''\leq\alpha\}}
{\#\{p_i' \leq \alpha\}}$
With dataset $D1$ there is nothing rare: the lower the significance level the smaller the proportion because it is harder to observe significance.
But when I plot it with dataset $D2$, I have this weird behavior in the plot: it kind of goes up with alpha, but then at some point it goes down with large alphas.
I can't figure out why this is happening. The same happens with other tests such as Wilcoxon and Sign. Any ideas?

 A: I spent some time thinking about this yesterday and haven't find any interesting explanation. I am not even sure I fully understand what you are doing and why.
However, I do have one idea about what you could do, namely double-check and triple-check your code and your data as it's very easy to let bugs slip through. Run the procedure with small toy examples where you can follow the computations manually, try other (possibly smaller) data sets and see if everything looks fine, run the random splitting/testing part at one error level and directly plot/inspect the resulting p-values, make sure your proportions really include as many cases as you think, look for strange things in your data sets (including missing values), generate random data sets with known characteristics (e.g. no effect at all or a predefined effect and normal error), run them through your procedure and check if the output looks sensible, etc. I assume you have already done some of that but maybe these remarks will suggest other things to try… 
