1
$\begingroup$

I've been doing research on the topic of multiple comparisons, and most of the definitions I came across with, had the words

multiple simultaneous statistical tests

in them. Can anybody define what simultaneous means?

For example, if I'm doing 3 separate differential gene expression experiments, at the same time; should I join all the obtained p-values together in order to have less false positives? This doesn't sound quite right to me.

$\endgroup$
1
  • $\begingroup$ I don't know anything about gene expressions. However if you for one observation have two values: e.g. heigh and weight of a person, then you could (and probably should) run tests on height and weight simultaneously, and not seperatly. This is what I perceive to be the general meaning of "simultaneous" in statistics. $\endgroup$ Commented Mar 28, 2018 at 9:11

1 Answer 1

1
$\begingroup$

Intuitively, the more you look at the same data in an attempt to find some thing or another, the more you are likely to find just by virtue of your multiple looks. This is why we need multiplicity adjustments - to make sure what we find isn't simply a function of our multiple looks at the same data.

The word simultaneous in this context doesn't necessarily mean that all of the multiple looks are taking place at the same time. What matters is that, when you draw the line after completing all analyses for the same data, you factor in the number of looks you had at the data and also whether or not those looks were planned before the data were collected or not. Some people believe that, if the looks were planned a priori, then there is no need for multiplicity adjustment across looks. But if the looks were not planned up front, multiplicity adjustment will be needed.

As an example, imagine you randomly assign patients to three treatments (A, B and C) and want to see if there is a difference between any of the treatments with respect to the mean value of some outcome variable Y. Then you might first perform a one-way ANOVA to find evidence of a difference between at least two of the treatments. Following this, you can perform multiple pairwise comparisons (i.e., B vs A, C vs A, C vs B) to determine which specific pairs of treatments may be different. These pairwise comparisons will have to be adjusted for multiplicity.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.