1
$\begingroup$

I have two datasets, $D1$ and $D2$, each containing many samples $S_i$ of 100 numbers, each sample from a different (unknown) population. For example:

$S_1=\{0.125, 0.122, 0.053, \dots\} \\ S_2=\{0.001, 0.013, 0.034, \dots\} \\ \dots $

The are both quite large. $D1$ contains almost 6M samples, and $D2$ contains almost 500K.

I randomly split each sample $S_i$ in two disjoint subsamples of size 50 each, let them be $S_i'$ and $S_i''$. For each subsample I run a t-test for the null hypothesis of mean equal zero, so I have two p-values: $p_i'$ and $p_i''$.

My question of interest is: how often do I observe significance in $S_i''$ if I observed significance in $S_i'$. So, for different significance levels, I plot this proportion:

$\displaystyle\frac {\#\{p_i' \leq \alpha~and~p_i''\leq\alpha\}} {\#\{p_i' \leq \alpha\}}$

With dataset $D1$ there is nothing rare: the lower the significance level the smaller the proportion because it is harder to observe significance. But when I plot it with dataset $D2$, I have this weird behavior in the plot: it kind of goes up with alpha, but then at some point it goes down with large alphas.

I can't figure out why this is happening. The same happens with other tests such as Wilcoxon and Sign. Any ideas?

enter image description here

Improve this question
$\endgroup$
6
  • 3
    $\begingroup$ I'm afraid I can't follow what you're doing. Perhaps you could illustrate with a tiny version of your data, showing us more clearly what you mean by a "dataset," what you mean by a "sample," exactly how you perform the "splitting," how you manage to obtain two p-values for each t-test, and what distinction (if any) you are making between "p-value," "significance," "significants," and "alpha." $\endgroup$
    – whuber ♦
    Jul 4, 2013 at 16:48
  • $\begingroup$ @whuber Edited the question $\endgroup$
    – Julián Urbano
    Jul 4, 2013 at 17:02
  • $\begingroup$ Do both data set have the same number of samples? $\endgroup$
    – Gala
    Jul 4, 2013 at 17:21
  • $\begingroup$ @GaëlLaurans No, but they're both quite large: D1 has almost 6M samples and D2 has almost 500K $\endgroup$
    – Julián Urbano
    Jul 4, 2013 at 17:26
  • 1
    $\begingroup$ Is your splitting procedure randomized or not? $\endgroup$
    – whuber ♦
    Jul 4, 2013 at 20:16

1 Answer 1

Reset to default
2
$\begingroup$

I spent some time thinking about this yesterday and haven't find any interesting explanation. I am not even sure I fully understand what you are doing and why.

However, I do have one idea about what you could do, namely double-check and triple-check your code and your data as it's very easy to let bugs slip through. Run the procedure with small toy examples where you can follow the computations manually, try other (possibly smaller) data sets and see if everything looks fine, run the random splitting/testing part at one error level and directly plot/inspect the resulting p-values, make sure your proportions really include as many cases as you think, look for strange things in your data sets (including missing values), generate random data sets with known characteristics (e.g. no effect at all or a predefined effect and normal error), run them through your procedure and check if the output looks sensible, etc. I assume you have already done some of that but maybe these remarks will suggest other things to try…

Improve this answer
$\endgroup$
4
  • $\begingroup$ I've been able to get x3 data and it looks the same, so I'd discard the sample size issue. I've also checked the code several times and nothing seems wrong. Made a run with random data as you suggested and it looked fine. So I don't know why that can happen...maybe it's a problem with the mechanism underlying data generation, maybe it's not as random as it should. Thanks anyway. $\endgroup$
    – Julián Urbano
    Jul 10, 2013 at 2:19
  • $\begingroup$ btw, I'm looking into this because the real data violates the tests' assumptions, so I'm trying to see which one is more robust with the actual data we have. $\endgroup$
    – Julián Urbano
    Jul 10, 2013 at 2:20
  • $\begingroup$ @Julián Urbano Tests are constructed to guarantee a specific error rate under the null. One way to check that with a simulation is to generate random data and see if the proportion of significant results is close to the nominal error level. Your data are different, you clearly have many significant results so you are looking at a strange power curve, not so much checking the test's control of the error level. $\endgroup$
    – Gala
    Jul 10, 2013 at 5:55
  • $\begingroup$ I know, but I can't generate real data where the null is true, so I can't check type I error rates. I'm looking at results in both subsets: lack of power, conflicts, etc. $\endgroup$
    – Julián Urbano
    Jul 10, 2013 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.