3
$\begingroup$

I have created a mixed distribution model comprising 80% $H_0$ plus 20% $H_1$ to illustrate the link between the expected proportions of true and false positives and negatives in the PDF, CDF and p-value distributions.

When the effect size is small (25% increase in expected correlations vs $H_0$), everything looks consistent:

p-Model 1

However, when I increase the effect size to 75% by separating $H_1$ from $H_0$, things don't seem to add up because I get apparent false negatives (grey area) in the p-value distribution, which are no longer there in the PDF and CDF distributions:

p-Model 2

Conversely, when I reduce the effect size to 0% (so that $H_1$ = $H_0$), the grey area disappears altogether from the p-value distribution, whereas it is even more prominent in the PDF and CDF distributions:

p-Model 3

Now, I'm sure I have done all the maths right and the actual distribution curves are correct and consistent, but perhaps I am misinterpreting the areas between these curves?

What do these four different coloured areas mean in each chart, and how do they relate to the expected proportions of true and false positives and negatives?

How can I show the link between these proportions in the three distributions, and why don't they correspond with each other as shown?

Please help me understand the apparent discrepancy, thanks!

Improve this question
$\endgroup$
13
  • $\begingroup$ Could you maybe explain what you were trying to do, mathematically, with those charts? I think I get the first two, but the one on the right has me scratching my head a little bit. Writing it out explicitly also might help diagnosing any issues in the logic. Also, what software did you use to make them? If that's the default styling, it's very handsome $\endgroup$
    – shadowtalker
    Commented Apr 6, 2016 at 12:20
  • $\begingroup$ Hi ssdecontrol. Yes, essentially I created this mixed distribution model to understand and illustrate the mathematical link between all these elements (PDF, CDF, p-values, alpha, effect size, as well as the expected proportions of true and false positives and negatives - both for my own benefit, and for the benefit of my colleagues (once I can reconcile this problem). I'm trying to illustrate these relationships graphically, because pictures are more intuitive for myself and other non-statisticians to understand. $\endgroup$
    – Kelvin
    Commented Apr 6, 2016 at 12:25
  • $\begingroup$ Ah, I understand the problem now. What happens when you run a simple cross-tab of the possibilities? It would be a good demonstration anyway. But the fact that charts six and four disagree so strongly suggests a coding error, and looking at a simple cross tab might help open up the guts of the issue $\endgroup$
    – shadowtalker
    Commented Apr 6, 2016 at 12:30
  • $\begingroup$ In principle, since you generated this data, you ought to know approximately how many should fall in each category $\endgroup$
    – shadowtalker
    Commented Apr 6, 2016 at 12:30
  • $\begingroup$ It's strange, because the p-value curves themselves seem correct, thus I suspect that my interpretation of the areas is wrong. I can post the model itself if it helps (and assuming I'm allowed to attach it)? $\endgroup$
    – Kelvin
    Commented Apr 6, 2016 at 12:31

1 Answer 1

Reset to default
0
$\begingroup$

So I have worked at this problem from the opposite end, by calculating the proportions of true and false positives and negatives directly from the CDF (rather than from the p-values) and then "normalizing" these proportions to the diagonal line that represents the p-values under the null hypothesis. From this, it is clear that these proportions do not correspond to the areas between the p-value distribution curves as I had previously thought:

p-Model 1 p-Model 2 p-Model 3

But that still leaves the question of what do the areas between the p-value curves represent, which I posed in a separate thread here:

What do the four coloured areas of this p-value distribution actually represent?

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.