0
$\begingroup$

If I conduct 19 tests (e.g. 19 pairwise comparisons of treatment means in an experiment), what is the probability that I will find at least one P value < 0.05 due to chance alone? With number of pairwise comparisons given by k*(k-1)/2 where k is the number of treatment levels, how does one calculate this?

$\endgroup$
1
$\begingroup$

I found an answer here, which says that the chance also depends on the level of significance you have set.

The chance of committing at least one Type I error for a set of comparisons can be estimated as c * .05, where c is the number of comparisons made. For our example with three conditions, we can make three comparisons, this would be 3 * .05 = .15, or a 15% chance of making at least one Type I error.

Since Type I error is getting p-value small even if it should not be.

In statistical hypothesis testing, a type I error is the rejection of a true null hypothesis (also known as a "false positive" finding or conclusion)

$\endgroup$
1
$\begingroup$

It depends on the correlation of your 19 tests and the distribution of your $p$-values. For independent tests and uniformly distributed $p$-values (see Why are p-values uniformly distributed under the null hypothesis? and comments) the chance for no $p$-value below 0.05 would be $$\prod_{i=1}^{19} P(p_i<0.05)=(0.05)^{19}$$ and therefore the answer to your question regarding the complementary event is $$1-(0.05)^{19}.$$

If the tests have a correlation of 1, the probability would only be 0.05.

If you don't know the correlation, the highest possible probability would be $19*0.05$.

With number of pairwise comparisons given by k*(k-1)/2 where k is the number of treatment levels, how does one calculate this?

  • If you have k treatment levels, you can compare the first one to k-1 other levels.
  • The next you compare to k-2 (because you already compared it to the first one).
  • The next after that to k-3.
  • And so on ... till you compare the last but one to the last.

So what's the sum from 1 to k-1? That's the well know formula from Gauß: $$\sum_{i=1}^{k-1}i=\frac{(k-1)k}{2}.$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.