When correcting for multiple comparisons, do you correct all p-values of all tests used from the same dataset? I have this pretty basic question, but I haven't found an answer. I have a dataset, and I've performed many statistical tests comparing various variables. Do I correct all p-values, regardless of the statistical tests used and the variables compared? Thanks!
 A: I’m making a bit of a guess at what you’re doing based on your comment, but it sounds like you want to hunt for a difference until you find one, and that’s a case where you not only control for the number of tests you do but also control for the number of tests you considered doing but didn’t do after you found the answer you wanted.
The reason to control for the number of tests that you would consider is to keep yourself from getting out of doing an adjustment for multiple comparisons of you happen to get lucky on the first test or two (or three, etc...).
This makes the most sense to me when you do tests like in post hoc ANOVA. Say there are five groups: A, B, C, D, and E. You want to find some evidence that not all are the same, so you keep testing them in pairs. AB is insignificant. AC is insignificant. Et cetera. But then BD winds up significant. You’re done, right? If B and D are different, then the five groups can’t be the same. And you even adjusted for the six comparisons.
No, you should adjust for all ten pairwise comparisons. It shouldn’t matter that you did the comparisons in some order. You planned to do all ten until you found an answer you liked.
So I’d say that you should be making some kind of adjustment for all of the comparisons.
