4
$\begingroup$

First time question on this site, so please bear with me, thank you:

I have 6 coin-flip-type experiments for which I can calculate 6 binomial p-values. I would now like to calculate the significance of observing at least 4 p-values < 0.05 in six experiments total.

In one approach, I used Fisher's method (http://en.wikipedia.org/wiki/Fishers_method) to compound the p-values, but I wanted to add an additional test based on simulating the original coin-flip data.

To this end, I performed random coin-flips (P=0.5) for each of the 6 experiments; the total number of flips differs between these 6 experiments but is irrelevant. I then count how many times (out of 100 simulations) the binomial p-value < 0.05. Simulating the original data 100 times, I get the following number of significant binomial p-values ("false positive hits") from these 6 experiments:

12, 13, 9, 10, 7, 11

Or divided by 100 (= frequency of false positive hits in simulated experiments):

0.12, 0.13, 0.09, 0.1, 0.07, 0.11

How do I calculate the probability that 4 or more out of these 6 would be positive given these frequencies? I realize that for calculating the probability that 6/6 are positive, I would simply multiply 0.12 x 0.13 x 0.09 x 0.1 x 0.07 x 0.11. But for 1-5/6 it's more complicated. I'm leaning towards a hypergeometric test, since I have to draw 6 times and I think there's no replacement, but I want to double-check with you experts. Thank you!

Improve this question
$\endgroup$
3
  • $\begingroup$ I perform N coin-flips where N = sample size in experiment (ie. 88). In R, I use the following: heads=sum(sample(x=c(0,1),size=88,replace=T,prob=c(0.5,0.5))) and tails = 88-heads $\endgroup$
    – reviewer3
    Commented Nov 7, 2013 at 4:18
  • $\begingroup$ And yes for each experiment, I am simulating from an assumed model of an even coin (P=0.5), but my number of trials = sample size of the original experiment. Maybe resampling is a misnomer? $\endgroup$
    – reviewer3
    Commented Nov 7, 2013 at 4:27
  • $\begingroup$ Yes, resampling means that in some fashion you draw samples from or based on the original data, not just its sample size. Replace 'resampled' with 'simulated' etc $\endgroup$
    – Glen_b
    Commented Nov 7, 2013 at 4:29

1 Answer 1

Reset to default
2
$\begingroup$

I have a number of additional comments to make regarding issues I have with what you're describing (which I will come back to), but first let's just deal with the simple question:

I would now like to calculate the significance of observing at least 4 p-values < 0.05 in six experiments total.

By 'significance' I assume you mean 'probability of ... under the null'.

Simple version:

Imagine that the sample sizes were such that we could treat the distribution of p-values under the null as continuous. In that case they will be uniform under the null.

Then under $H_0$ each experiment has a 5% chance of giving a p-value below 0.05

The distribution of the number of experiments yielding p-values below 0.05 under the null is $\text{binomial}(6,0.05)$

Let $X\sim \text{binomial}(6,0.05)$. Then $P(X\geq 4) = 8.64\times 10^{-5}$

Less simple version:

The distribution of p-values is discrete, and a significance level of exactly 0.05 won't typically be attainable. A more accurate answer in this case involves finding the largest possible p-value less than 0.05 (which depends on the exact sample size for each experiment), and then doing a similar calculation for that case. It will give a smaller probability than the one I just calculated. This involves some slightly more complicated calculation, but it's perfectly possible to do it exactly, without simulation.

(I don't think your simulation approach looks right, by the way, but since it's possible to do this question without worrying about that, I won't labor the point.)

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you for your answer. I'll have to think more about this in the morning, but in the meantime I implemented your 'simple version', except that I calculated binomial (6, P) with P = sum(12, 13, 9, 10, 7, 11)/600 rather than P=0.05 (although I understand why you chose 0.05). But wouldn't the distribution of p-values be discrete, since my simulated input data is by definition integers and I'm doing a finite number of simulations? Couldn't I use your 'simple' version as an approximation and simply state 'aggregated p-value < ...'? Thank you! $\endgroup$
    – reviewer3
    Commented Nov 7, 2013 at 7:46
  • $\begingroup$ Also, please let me know why you think my simulation approach doesn't look right. I plotted heads & tails after 10000 simulations and they look like a normal distribution centered around 0.5*(sample size). After only 100 simulations a bit more wobbly but close. Thank you! $\endgroup$
    – reviewer3
    Commented Nov 7, 2013 at 7:49
  • $\begingroup$ PS. I can't upvote your answer yet until someone else upvotes my question $\endgroup$
    – reviewer3
    Commented Nov 7, 2013 at 7:54
  • $\begingroup$ The appearance of the distribution of the number of heads wasn't the concern, but what you did with them. I'll try to come back with an explanation of what I think the problem was. $\endgroup$
    – Glen_b
    Commented Nov 7, 2013 at 10:27
  • $\begingroup$ Ok, thank you @Glen_b For calculation of real and simulated p-values in R I use pbinom(min(heads,tails),size=88,prob=0.5) if heads and tails were drawn from a sample size of 88. $\endgroup$
    – reviewer3
    Commented Nov 7, 2013 at 22:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.