MULTIPLE TESTING: Probability of specific outcomes from MULTIPLE experiments using randomized data

Question

This is a follow-up question to this one: Significance test across multiple simulated experiments There's one answer I'm leaning towards accepting, but I wanted to make sure I understood how significance can be calculated across multiple experiments OR estimated from a set of simulated experiments.

I have 6 datasets representing millions of coin-flip type experiments over thousands of samples, in which some samples MAY be non-randomly distributed; i.e. cumulative binomial prob < 0.05 in 1,2,3,4,5 or all 6 datasets.

Because I'm concerned about multiple testing in this setting, I want to know: How often do these samples score a binomial p value < 0.05 in 1,2,3,4,5 or all 6 simulated datasets (where I randomly flip the coin 100 times for each sample).

Simple question: How would you approach this question? Also, please let me know if I should be more specific or ask the question in a different way. Thank you!

UPDATE: Here's a specific example:

heads/tails:
exp. 1: 88/11, p < 0.05
exp. 2: 38/12, p < 0.05
exp. 3: 115/3, p < 0.05
exp. 4: 39/47, p > 0.05
exp. 5: 70/13, p < 0.05 
exp. 6: 33/30, p > 0.05

4 out of 6 experiments show a binomial prob < 0.05, the other 2 above. Note that the total number of coin tosses differs between experiments. Although I could multiply the six individual p-values to calculate an overall probability of observing these 6 results, I want for each experiment to count EQUALLY, independently of the total number of coin tosses. That's important, because the number of "coin tosses" in the actual data can differ by orders of magnitude!

Equally important, I'm concerned about multiple testing. I have > 30,000 samples in each experiment. If I have an overall p-value cutoff of 0.01, i will make 300 incorrect observations!

That's why I wanted to simulate each of the 6 experiments 100 times over, with their number of coin tosses = the original data:

heads/tails with fair coin (Pr=0.5):
exp. 1: 99 tosses, observed p < 0.05 in 100 simulations = 12
exp. 2: 50 tosses, observed p < 0.05 in 100 simulations = 13
exp. 3: 118 tosses, observed p < 0.05 in 100 simulations = 9
exp. 4: 86 tosses, observed p < 0.05 in 100 simulations = 10
exp. 5: 83 tosses, observed p < 0.05 in 100 simulations = 7 
exp. 6: 63 tosses, observed p < 0.05 in 100 simulations = 11

So, how would you calculate or use the simulated data to estimate the likelihood of observing the original 6 results by chance?

Answer 1

Stephan, I don't think that your question is very clear, perhaps because you are not clear on what estimation and significance represent.

Significance tests provide a p-value that is a measure of how extreme the results are relative to the expected distribution of results under the null hypothesis (i.e. the sampling distribution). The p-value is usually spoken of as the 'significance level' of the test. You determine the p-value by calculation.

The estimation part usually relates to determination of the value of a parameter of interest. For coin tossing that would be the long-run proportion of heads, the probability of the coin turning up heads on each toss. You estimate that parameter using the mean of the data.

The p-values that you calculate serve to tell you how unusual the observed result would be if the null hypothesis (usually Pr(heads)=0.5) was true. The level of significance in that is calculated, not estimated.

Now, in your experiment are you setting or assuming that Pr(heads) is the same each time? (Is it the same coin being tossed in each experiment?) If yes, then the best estimate of Pr(heads) is provided by the total fraction of the tosses that came up heads in all of the experiments.

If I can assume that the 'experiments' use different coins that may have different Pr(heads) then the probability of obtaining a p-value less than the arbitrary value of 0.05 depends on how far Pr(heads) deviates from 0.5 in how many of the coins.