I have created a mixed distribution model comprising 80% $H_0$ plus 20% $H_1$ to illustrate the link between the expected proportions of true and false positives and negatives in the PDF, CDF and p-value distributions.

When the effect size is small (25% increase in expected correlations vs $H_0$), everything looks consistent:

p-Model 1

However, when I increase the effect size to 75% by separating $H_1$ from $H_0$, things don't seem to add up because I get apparent false negatives (grey area) in the p-value distribution, which are no longer there in the PDF and CDF distributions:

p-Model 2

Conversely, when I reduce the effect size to 0% (so that $H_1$ = $H_0$), the grey area disappears altogether from the p-value distribution, whereas it is even more prominent in the PDF and CDF distributions:

p-Model 3

Now, I'm sure I have done all the maths right and the actual distribution curves are correct and consistent, but perhaps I am misinterpreting the areas between these curves?

What do these four different coloured areas mean in each chart, and how do they relate to the expected proportions of true and false positives and negatives?

How can I show the link between these proportions in the three distributions, and why don't they correspond with each other as shown?

Please help me understand the apparent discrepancy, thanks!

  • $\begingroup$ Could you maybe explain what you were trying to do, mathematically, with those charts? I think I get the first two, but the one on the right has me scratching my head a little bit. Writing it out explicitly also might help diagnosing any issues in the logic. Also, what software did you use to make them? If that's the default styling, it's very handsome $\endgroup$ Apr 6 '16 at 12:20
  • $\begingroup$ Hi ssdecontrol. Yes, essentially I created this mixed distribution model to understand and illustrate the mathematical link between all these elements (PDF, CDF, p-values, alpha, effect size, as well as the expected proportions of true and false positives and negatives - both for my own benefit, and for the benefit of my colleagues (once I can reconcile this problem). I'm trying to illustrate these relationships graphically, because pictures are more intuitive for myself and other non-statisticians to understand. $\endgroup$
    – Kelvin
    Apr 6 '16 at 12:25
  • $\begingroup$ Ah, I understand the problem now. What happens when you run a simple cross-tab of the possibilities? It would be a good demonstration anyway. But the fact that charts six and four disagree so strongly suggests a coding error, and looking at a simple cross tab might help open up the guts of the issue $\endgroup$ Apr 6 '16 at 12:30
  • $\begingroup$ In principle, since you generated this data, you ought to know approximately how many should fall in each category $\endgroup$ Apr 6 '16 at 12:30
  • $\begingroup$ It's strange, because the p-value curves themselves seem correct, thus I suspect that my interpretation of the areas is wrong. I can post the model itself if it helps (and assuming I'm allowed to attach it)? $\endgroup$
    – Kelvin
    Apr 6 '16 at 12:31

So I have worked at this problem from the opposite end, by calculating the proportions of true and false positives and negatives directly from the CDF (rather than from the p-values) and then "normalizing" these proportions to the diagonal line that represents the p-values under the null hypothesis. From this, it is clear that these proportions do not correspond to the areas between the p-value distribution curves as I had previously thought:

p-Model 1 p-Model 2 p-Model 3

But that still leaves the question of what do the areas between the p-value curves represent, which I posed in a separate thread here:

What do the four coloured areas of this p-value distribution actually represent?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.