5
$\begingroup$

Let's say that I have a group of subjects that were measured in two conditions (A and B) and I want to check three hypotheses: i) do data from A and B differ from each other?, ii) do data from A differ from zero?, iii) do data from B differ from zero?. So, my tests would be: a paired t-test for hypothesis i), and two simple t-tests for hypotheses ii) and iii).

I'm wondering whether I should or I shouldn't correct the alpha level of these three tests. I'm inclined to apply the correction for multiple comparisons to avoid type I errors, but at the same time I would also like to avoid type II errors.

$\endgroup$

2 Answers 2

1
$\begingroup$

Independently of the specifics of the tests you are considering, a general solution is to perform a power analysis. Set your error level at .05/3 and see what power you would attain with the effect you can expect and the sample size you intend to use. If power is too low/risk of type II error is too high for your taste, you have two (and a half) options:

  • Increase sample size
  • Give up multiple comparisons correction/increase error level (there is nothing sacred about 5% and nothing special about the correction for multiple comparisons, you can always set a higher error level to increase power; of course, you also need to get away with it publication-wise if that's your objective)

The half option:

  • Run another experiment (use measures with higher reliability, study a more homogeneous sub-population, make the experimental design within-subject, create a stronger effect by using another manipulation, etc.) or give up altogether (If there is no reasonable chance to identify a plausible effect, why spend time and money on it? Even if you do find a “significant” effect, it's likely to be an gross overestimation of the actual magnitude of the phenomenon.)

EDIT (following comment)

The important thing to understand is that power depends on alpha (the type I error rate), the sample size and the magnitude of the effect. If costs don't matter, you can always use an adjusted alpha and still attain the same level of power by increasing the sample size, thus making the problem disappear.

Assuming a fixed sample size, any decision regarding corrections for multiple comparisons will depend on what levels of type I and type II errors are acceptable. The trouble with any “standard practice” in this respect is that it wouldn't take the trade-off between the two into account. If the effect size is huge, a conservative type I error level will not make power unacceptably low. If the effect is tiny, sticking religiously to the 5% error level means giving type I errors enormous weight over type II errors.

It does not make sense not to factor this into the decision. Arguments about this problem (e.g. the call not to use corrected tests in epidemiology) rely on implicit assumptions about effect sizes and the relative costs of type I and type II errors (e.g. not identifying risk factors vs. falsely flagging something as risky).

$\endgroup$
5
  • $\begingroup$ I fully agree with what you expressed in your answer, but I'm still looking for a sort of "standard" answer. For instance, everyone would say that if you run an ANOVA with one factor (three levels) and you find that the factor is significant, you should proceed with multiple comparisons between levels and correct your alpha level. So, what would a "standard" person say about my example? That I should or shouldn't correct the alpha level? $\endgroup$
    – vikkor
    Commented Jun 14, 2013 at 13:16
  • $\begingroup$ Not everyone would say that, I generally wouldn't and would rather look at some plots of the data rather than do post-hoc tests. Really, what count as “standards” depends on the discipline and is not always the best way to approach a problem. Statistical analysis is not merely a matter of finding the “right” test and making a binary decision (“real” effect vs. no effect). $\endgroup$
    – Gala
    Commented Jun 14, 2013 at 13:25
  • $\begingroup$ That said, you can correct the alpha level and still attain an arbitrarily high level of power, it's just going to cost you. When people say, e.g. that a Bonferroni correction is conservative, it just means that it wastes resources and increases type II error rate for the same sample size. It does not automatically entail a higher type II error rate since you can always get back to the original level of power by increasing the sample size. $\endgroup$
    – Gala
    Commented Jun 14, 2013 at 13:31
  • $\begingroup$ That's why a standard answer is not very useful, it's just not possible to say that you should or shouldn't correct without knowing what the effect size is, how much each type of error would cost you and how much you are ready to spend for your experiment. $\endgroup$
    – Gala
    Commented Jun 14, 2013 at 13:43
  • $\begingroup$ @vikkor I expanded my answer to add a few remarks about this. $\endgroup$
    – Gala
    Commented Jun 14, 2013 at 14:15
1
$\begingroup$

When considering whether to correct for multiple comparisons I like to think about how my conclusions might change if I generated a bunch of completely random noise, gave it a plausible sounding name, and added it to my analysis.

Here are 2 extreme (and oversimplified) cases to use as example.

Case 1: I am fishing for anything significant, I have thrown a bunch of variables in my model that I think may possibly have a relationship with my outcome and if any of them are significant then I will claim success.

In this case if I start adding in random noise variables and don't adjust for multiple comparisons, then by chance alone I will find something significant if I add enough noise variables. So I should adjust here.

Case 2: I am very interested in the answer about A, but while doing the experiment I collect additional information as well (the added cost is minimal and could be useful for future research).

Here if a noise variable C is significant it does not (or at least should not) change my conclusions about A. So here I would not adjust.

Most cases will probably fall between the above, but it is a useful thought process. You can always compute both and look at the adjusted and unadjusted values. If they agree then there is really no problem, if one set shows significance but the other does not, then you can still report both and let the reader decide (or go back and collect more data to better answer the question).

For your case I think I would start with an overall test that simultaneously tests A=B=0, if that is not significant then it is probably not worth looking at the individual tests. If it is significant then you can look at the individual tests, this is called a protected test.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.