MULTIPLE TESTING: Probability of specific outcomes from MULTIPLE experiments using randomized data

This is a follow-up question to this one: Significance test across multiple simulated experiments There's one answer I'm leaning towards accepting, but I wanted to make sure I understood how significance can be calculated across multiple experiments OR estimated from a set of simulated experiments.

I have 6 datasets representing millions of coin-flip type experiments over thousands of samples, in which some samples MAY be non-randomly distributed; i.e. cumulative binomial prob < 0.05 in 1,2,3,4,5 or all 6 datasets.

Because I'm concerned about multiple testing in this setting, I want to know: How often do these samples score a binomial p value < 0.05 in 1,2,3,4,5 or all 6 simulated datasets (where I randomly flip the coin 100 times for each sample).

Simple question: How would you approach this question? Also, please let me know if I should be more specific or ask the question in a different way. Thank you!

UPDATE: Here's a specific example:

heads/tails:
exp. 1: 88/11, p < 0.05
exp. 2: 38/12, p < 0.05
exp. 3: 115/3, p < 0.05
exp. 4: 39/47, p > 0.05
exp. 5: 70/13, p < 0.05
exp. 6: 33/30, p > 0.05


4 out of 6 experiments show a binomial prob < 0.05, the other 2 above. Note that the total number of coin tosses differs between experiments. Although I could multiply the six individual p-values to calculate an overall probability of observing these 6 results, I want for each experiment to count EQUALLY, independently of the total number of coin tosses. That's important, because the number of "coin tosses" in the actual data can differ by orders of magnitude!

Equally important, I'm concerned about multiple testing. I have > 30,000 samples in each experiment. If I have an overall p-value cutoff of 0.01, i will make 300 incorrect observations!

That's why I wanted to simulate each of the 6 experiments 100 times over, with their number of coin tosses = the original data:

heads/tails with fair coin (Pr=0.5):
exp. 1: 99 tosses, observed p < 0.05 in 100 simulations = 12
exp. 2: 50 tosses, observed p < 0.05 in 100 simulations = 13
exp. 3: 118 tosses, observed p < 0.05 in 100 simulations = 9
exp. 4: 86 tosses, observed p < 0.05 in 100 simulations = 10
exp. 5: 83 tosses, observed p < 0.05 in 100 simulations = 7
exp. 6: 63 tosses, observed p < 0.05 in 100 simulations = 11


So, how would you calculate or use the simulated data to estimate the likelihood of observing the original 6 results by chance?

• Your edit has not really made your question clear. The p-value is not a 'binomial prob', Pr(heads). A sample is a collection of observations, but what an 'experiment' is depends on context and design. If you have 30,000 observations in a sample there is a 1 in 20 chance that the p-value will be less than 0.05 if the null is true (and the test assumptions are valid). When you ask about 'observing the original 6 results' do you mean the proportions of heads or the 'significant'/'not significant' results? How is Glen_b's answer to the other question not sufficient for this? – Michael Lew - reinstate Monica Nov 11 '13 at 23:06
• Thank you for taking a look, @Michael Lew I'm sorry I'm getting tripped up by the language, I'm really trying to clarify this. Please forget about samples for now. I'm only looking at 1 example across 6 experiments (or datasets). When I ask about 'observing the original 6 results', I mean the significant/not significant results, i.e. 4/6 significant. Glen_b 's answer for this could be sufficient, but is this the only way? What if these probabilities weren't based on a binomial but for example on a normal distribution? How would you compound a set of calculated probabilities? Fisher's method? – reviewer3 Nov 11 '13 at 23:19
• By Fisher's method I assume you mean his method for combining p-values. You can't use that on significant/not significant results because you need the exact p-values (i.e. you need significance test results, not hypothesis test outcomes). There are many different methods to combine p-values, for example, see Whitlock (J. EVOL. BIOL. 18 (2005) 1368–1373). If you don't understand the distinction between hypothesis testing and significance tests (and many don't) then see my paper ncbi.nlm.nih.gov/pubmed/22394284 for an approachable explanation. – Michael Lew - reinstate Monica Nov 12 '13 at 0:06
• Sorry, wrong link to my paper. This is the full text: ncbi.nlm.nih.gov/pmc/articles/PMC3419900 – Michael Lew - reinstate Monica Nov 12 '13 at 0:07
• @MichaelLew Thank you for the references! I read them both and I feel quite fortunate that you saw my post. Having read your paper, I must agree that I'm attempting to combine incompatible approaches without even being fully aware of their distinction. Let me first address Fisher's & Stouffers: I tried combining the p-values produced from the binomial test and found that Fisher's method was "too sensitive", exactly as described by Whitlock: "Fisher’s method is asymmetrically sensitive to small P-values compared to large P-values." I then tried an unweighted Stouffer's which improved results. – reviewer3 Nov 12 '13 at 1:49

Stephan, I don't think that your question is very clear, perhaps because you are not clear on what estimation and significance represent.

Significance tests provide a p-value that is a measure of how extreme the results are relative to the expected distribution of results under the null hypothesis (i.e. the sampling distribution). The p-value is usually spoken of as the 'significance level' of the test. You determine the p-value by calculation.

The estimation part usually relates to determination of the value of a parameter of interest. For coin tossing that would be the long-run proportion of heads, the probability of the coin turning up heads on each toss. You estimate that parameter using the mean of the data.

The p-values that you calculate serve to tell you how unusual the observed result would be if the null hypothesis (usually Pr(heads)=0.5) was true. The level of significance in that is calculated, not estimated.

Now, in your experiment are you setting or assuming that Pr(heads) is the same each time? (Is it the same coin being tossed in each experiment?) If yes, then the best estimate of Pr(heads) is provided by the total fraction of the tosses that came up heads in all of the experiments.

If I can assume that the 'experiments' use different coins that may have different Pr(heads) then the probability of obtaining a p-value less than the arbitrary value of 0.05 depends on how far Pr(heads) deviates from 0.5 in how many of the coins.

• thank you, and yes, my use of 'statistics language' lacks precision. I understand that the p-value for 1 sample in 1 experiment is calculated not estimated. I also understand that if I want to calculate the likelihood a specific outcome (p = 0.05) is observed in 3 experiments, I would have to multiply (i.e. .05 x .05 x .05). But at the heart of my question is: How do I calculate the likelihood of having 2 out of 3, or 4 out of 6 experiments with a p-value < 0.05? – reviewer3 Nov 11 '13 at 21:46
• Because I didn't know the answer to this question, I decided to simulate the data, by repeating each experiment with a fair coin (Pr=0.5) to give me an estimate of how frequently 1,2,3,4,5 or 6 out of six times the coin tosses result in a distribution that differs significantly from the expectation. I will update my question above with a specific example, as I agree that I've been unclear. But to answer your question, yes, my expectation is a fair coin (Pr = 0.5) and the number of tosses is sample & experiment-specific. – reviewer3 Nov 11 '13 at 21:53