and thanks for taking the time to look at this thread!
I'm massively stumped about exactly when it's 'correct' to be correcting for multiple comparisons by adjusting the threshold p-value. I know in post-hoc tests after an omnibus test, it's absolutely the 'done thing', and I can see why: by reducing the threshold p-value, you reduce the chance of making a Type-I error, trying to make it so you still only have a 5% chance (when p≤0.05) of getting a false-positive, thus adjusting for the number of tests you run. But why is this logic not extended to all tests in an article?
My research (and specific problem)
To give some background of the design I'm using (and hence why I'm stumped): we have 8 questions, all which are asked for 4 'categories, and within each 'category', we have 2 stimuli which participants all rate simultaneously. The questions are entirely different, and we have different hypotheses for what should happen with each, so the scores can't be combined in any way. Thus, we're running Wilcoxon's Signed Ranks test (the data is, unfortunately, non-parametric) between each set of stimuli for all categories and questions: thus, we have 32 tests.
Due to the sheer volume of the tests we're running, it seems highly likely that at least one will be a 'false-positive' when p≤0.05. That suggests to me that the p-value should be reduced to control for these multiple comparisons.
But then, each tests a functionally discrete hypothesis, so why should we have to? You wouldn't do that if you ran several ANOVA-type omnibus tests, or if you ran any number of different tests on the same results (e.g., regression and factor analysis, as a relatively weak example).
And to add further confusion, this is one of four studies of relatively similar design. Surely, if we want to reduce finding any 'false-positives', we should control the p-value across all tests conducted before writing up the article?
So, as you can see, I'm pretty stumped. I realise controlling for all comparisons in a paper seems ridiculous, and must be wrong, but it follows all the logic I've seen so far about why we correct for multiple comparisons in the first place. But I'm fully prepared to accept I'm an idiot!