3
$\begingroup$

I have 10 data sets with normal distributions and carrying out significance tests on each one. With this many data sets, there is a good chance that some will be significant by chance. I am going to try the Bonferroni method to solve this problem but another approach that occurred to me is a binomial one.

i.e. if the p value for each individual significance is 0.01 then the probability of getting a significant result follows B(10, 0.01). You can then generate the probability of any number of significant results. Is this method "legit"? I can't see anything wrong with it and it has been mentioned in another thread but does not seem to be very common and woud like to know if there is an issue with using it.

$\endgroup$
3
$\begingroup$

If you're interested in some overall sense of significance, then yes you could combine independent tests with constant $\alpha$ using the binomial. *

However, that's a somewhat different aim from adjusting significance to control the overall type I error rate in a number of individual tests.

* The reliance on whether individual tests achieve some particular $\alpha$ is somewhat arbitrary. To get some overall idea of signficiance, it would instead be more common to combine the p-values, for example via Fisher's method.

$\endgroup$
  • $\begingroup$ (+1) Such "binomial" method is sometimes applied in neurophysiological studies, where people record a bunch of neurons in several conditions, and then for each neuron test if it is "tuned" to condition, i.e. if its firing rate significantly depends on condition. This is usually reported like e.g. "137 out of 978 neurons were significantly tuned with p<0.05", and sometimes a binomial test is performed to demonstrate that 137 out of 978 is a significant number (as an average 0.05*978=49 would be expected under null). I have never seen Fisher's or Stouffer's method applied in such a situation. $\endgroup$ – amoeba says Reinstate Monica Aug 16 '14 at 13:57
  • $\begingroup$ @amoeba Thanks. In that particular case you're talking about roughly a thousand tests, rather than the 10 in the question. With n=10, you might (relatively easily) not get any significant results even though a Fisher test would strongly reject (p-values unusually low as a collection, but none happened to be below 0.05). But with 1000 tests it would be weird if you didn't get dozens of them even if the null were true. It's also possible that it's not as important to wring every shred of power out of it in that case. $\endgroup$ – Glen_b -Reinstate Monica Aug 16 '14 at 23:58
  • $\begingroup$ e.g. (0.0627, 0.4855, 0.1789, 0.0931, 0.7950, 0.0836, 0.4894, 0.0973, 0.0914, 0.3796) has no p-values below 0.06, yet Fisher's method gives p=0.0304 $\endgroup$ – Glen_b -Reinstate Monica Aug 17 '14 at 0:08
  • $\begingroup$ Thanks. Exactly what I needed (and even an example to check I am doing Fisher's method right) $\endgroup$ – user39707 Aug 17 '14 at 9:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.