This is a follow-up question to this one: Significance test across multiple simulated experiments There's one answer I'm leaning towards accepting, but I wanted to make sure I understood how significance can be calculated across multiple experiments OR estimated from a set of simulated experiments.
I have 6 datasets representing millions of coin-flip type experiments over thousands of samples, in which some samples MAY be non-randomly distributed; i.e. cumulative binomial prob < 0.05 in 1,2,3,4,5 or all 6 datasets.
Because I'm concerned about multiple testing in this setting, I want to know: How often do these samples score a binomial p value < 0.05 in 1,2,3,4,5 or all 6 simulated datasets (where I randomly flip the coin 100 times for each sample).
Simple question: How would you approach this question? Also, please let me know if I should be more specific or ask the question in a different way. Thank you!
UPDATE: Here's a specific example:
heads/tails: exp. 1: 88/11, p < 0.05 exp. 2: 38/12, p < 0.05 exp. 3: 115/3, p < 0.05 exp. 4: 39/47, p > 0.05 exp. 5: 70/13, p < 0.05 exp. 6: 33/30, p > 0.05
4 out of 6 experiments show a binomial prob < 0.05, the other 2 above. Note that the total number of coin tosses differs between experiments. Although I could multiply the six individual p-values to calculate an overall probability of observing these 6 results, I want for each experiment to count EQUALLY, independently of the total number of coin tosses. That's important, because the number of "coin tosses" in the actual data can differ by orders of magnitude!
Equally important, I'm concerned about multiple testing. I have > 30,000 samples in each experiment. If I have an overall p-value cutoff of 0.01, i will make 300 incorrect observations!
That's why I wanted to simulate each of the 6 experiments 100 times over, with their number of coin tosses = the original data:
heads/tails with fair coin (Pr=0.5): exp. 1: 99 tosses, observed p < 0.05 in 100 simulations = 12 exp. 2: 50 tosses, observed p < 0.05 in 100 simulations = 13 exp. 3: 118 tosses, observed p < 0.05 in 100 simulations = 9 exp. 4: 86 tosses, observed p < 0.05 in 100 simulations = 10 exp. 5: 83 tosses, observed p < 0.05 in 100 simulations = 7 exp. 6: 63 tosses, observed p < 0.05 in 100 simulations = 11
So, how would you calculate or use the simulated data to estimate the likelihood of observing the original 6 results by chance?