2
$\begingroup$

I know that if there is significant difference in means among groups in ANOVA, then one can perfrom post-hoc test ( Tukey HSD or something similiar) to get to know within which groups the difference is significant. Then we have to remember about multiple comparisons procedures and p-value corrections.

I wonder, is this situation is a multiple comparisons problem:

I have 7 variables, each measured before and after treatment. Now I want to perform repeated measures ANOVA for each variable separetly grouped by treatment (before and after). There will be 7 tests -> 7 hypothesis. Is that a situation where I should remeber about p-value corrections or not? Is is a multiple comparisons situation?

Thanks for any comment.

Improve this question
$\endgroup$
1
  • $\begingroup$ I think most people would not do a multiplicity correction because they are separate variables. However in a case where you are, say, selecting which variable is best, then you would, because then in essence you are comparing the variables. $\endgroup$
    – Russ Lenth
    Oct 3, 2014 at 12:47

2 Answers 2

Reset to default
2
$\begingroup$

It depends on whether you want to control for the overall type I error. My guess would be yes. Multiple testing arises when you want to make sure in all 7 tests, the probability of at least one false negative is less than 5%.

However in my opinion, multiple testing adjustment is less useful in practice as it has low power. When you conduct a lot of comparisons (7 for example), you would end up with insignificant results no matter which method is used for pvalue adjustment.

To make things even worse, multiple testing happens not just in ANOVA but also in regression. Say, you fit a linear regression with 6 covariates. In the end, you will get 7 pvalues (extra one for intercept).The pvalue is calculated by comparing the estimated coefficient with zero so we actually conducted 7 comparisons. In theory, we should perform multiple testing correction. Have you seen anyone doing so? Not to mention all those model selection procedures to pick up 2 significant predictors from a pool of 50 covariates. You count how many tests are done.

It is bad that no textbook covers this. Recently there starts to have some discussion within stats society regarding this issue. No sure if there is a good solution......

My suggestion to you is to ignore this when there are too many groups. This does NOT mean you are manipulating your experiment to report a wrong pvalue that favours your conclusion. It simply means:

You cannot effectively control overall type I error in multiple comparison. However you can still control for the individual type I errors

It is valid to ignore multiple comparisons as long as you correctly interpret your pvalues.

Peter

Improve this answer
$\endgroup$
4
  • 3
    $\begingroup$ I disagree with the advice to bypass multiplicity adjustments when there are too many groups. It says in essence to pretend an issue isn't there when it becomes too burdensome to address it. $\endgroup$
    – Russ Lenth
    Oct 3, 2014 at 12:43
  • 1
    $\begingroup$ Not really. It says there are certain questions (overall type I error) you simply cannot answer. However, you can still answer easier questions (individual type I error). $\endgroup$
    – Peter
    Oct 3, 2014 at 13:06
  • 1
    $\begingroup$ Yes, really. I agree that it's important to explain what adjustment was or was not made. But if there is a multiplicity issue and it hasn't been addressed, then the analysis is deficient. A lot of work in, e.g., gemonics research involves huge numbers of tests, and they don't ignore multiplicity issues. $\endgroup$
    – Russ Lenth
    Oct 3, 2014 at 14:29
  • 1
    $\begingroup$ Let's say apple sales 20 million iphone 6 in the first 2 weeks. Multiple comparison will help to answer the question "How likely that at least one of the iphone is deficient?". You should be able to answer that question without calculation. In some cases, controlling overall error rate is meaningless. You have to focus on the individual test. It is not that the analysis is deficient. It is just that a particular type of questions bear no practical value. $\endgroup$
    – Peter
    Oct 7, 2014 at 12:56
0
$\begingroup$

It depends on what you are trying to do.

Are you trying to do model selection to to either 1) Describe the data or 2) Predict future data?

In either case you should look into model selection measures (AIC, BIC, Mallows Cp, etc). These procedures aren't inferential, don't produce p-values, and so the multiple comparison question becomes moot.

Or are you trying to do inference? In this case I would suggest correcting for multiple comparisons.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.