5
$\begingroup$

For my data, I need to run numerous linear regression analyses, separately for each of my two groups of subjects as well as different conditions. Together, I would have to run approximately 40 regression analyses across the 3 conditions for each group. I'm aware that we should correct for multiple comparisons for the regression coefficients within a model. But in my case, should I also do a similar correction for the p-value associated with the regression model itself (ie: the F test)?

For example, if in one condition, I have to run 25 regression analyses for each group, would it make sense to divide alpha=.05 by 25?

I've been going through different websites and textbooks, but can't seem to find an answer. Any suggestions/help would be appreciated!

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ It depends on whether you want to control your overall type I error rate at some particular level or not. On the whole I think people worry about this much more often than strictly necessary, but sometimes it matters. $\endgroup$
    – Glen_b
    Commented Apr 1, 2013 at 1:11
  • 1
    $\begingroup$ @Glen_b Some people don't worry at all. :-) $\endgroup$
    – whuber ♦
    Commented Apr 1, 2013 at 13:57

2 Answers 2

Reset to default
8
$\begingroup$

As Glen says, it depends on your hypotheses, and especially the relative importance of Type I vs Type II errors. For example, if you're testing a new drug, you don't want to risk missing side-effects because they were rendered nonsignificant by the multiple comparison correction. A similar principle applies when testing for effects of contaminants in the environment. If you're data mining without strong a priori expectations, you might be more inclined to control the Type I error rate to ensure that any 'interesting' results you find are genuine.

Bonferroni in particular has fallen out of favour because it is very conservative (i.e. by controlling false positives you're drastically increasing the number of false negatives). Other options are the Sidak correction (which is less conservative when the family of comparisons is large), or my favourite, a false discovery rate (FDR) correction using some variant of the Benjamini-Hochberg procedure.

A family of comparisons (hypotheses) is difficult to define, but a loose definition is any group of comparisons in which you expect the difference to be in the same direction. For example, several alternative correlated measures of plant growth would certainly constitute a family of comparisons that should be corrected.

With 25 comparisons, you probably should do some correction. If you're using R, check out the fdrtool or qvalue packages. qvalue is especially easy, because it allows you to input a string of p-values from your 25 comparisons, and returns a list of 25 q-values (significance controlled for FDR).

In any case, make sure you include effect sizes (r, R^2, d). Effect sizes with confidence intervals are much more informative than arbitrary significance cutoffs.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for your response! That's very helpful. For my analyses, I do anticipate to see some relationships but lack previous empirical evidence to set specific hypotheses. So I should probably be more inclined to control the Type I error rate in my case then. I also came across other sources that suggested a less conservative approach in correcting for 'multiple comparisons'. I will look into Sidak correction and FDR correction. Thanks again! I really appreciate it! $\endgroup$
    – Julia
    Commented Apr 3, 2013 at 4:34
  • $\begingroup$ No worries! If you don't have clear empirically-based expectations then yes, I think you're right to use Sidak/FDR. A good option I think is to give the original p-values, and then indicate (e.g. bold font or asterisks) whether the comparison was significant after the correction. That way the reader can decide for themselves how much they trust the result. $\endgroup$
    – atrichornis
    Commented Apr 4, 2013 at 9:17
0
$\begingroup$

In order to correctly control your type 1 error you do need to make some sort of adjustment. Are your hypotheses nested in any way? If so you can take advantage of this structure (i.e. closed hypothesis testing). Also, if you think that groups of hypotheses are correlated, a much less conservative adjustment is a Benjamini-Hochberg step-up or step-down procedure.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Yes I believe so. That was another question I had in mind, so thanks for bringing this up! If say my 25 univariate regression analyses can be grouped under 5 general categories, then would I divide alpha=.05 by 5? (based on Bonferroni's correction for now before I look up Benjamini-Hochberg correction). Would you also be aware of any relevant methods/stats papers that I can cite to justify my correction? Thanks for your help! $\endgroup$
    – Julia
    Commented Apr 3, 2013 at 4:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.