Alpha correction in a family of tests Suppose that one runs 8 correlation analyses but in two separate batches: Would the alpha-correction apply to all 8 tests as a family or each of the 4 tests as a family?
For example, if there are two research questions and both require correlation analysis, the researcher conducts the first 4 tests (for research question 1) and then conducts the second 4 tests (for research question 2).
In this case, should the alpha be corrected for a family of 4 tests or for a family of 8 tests?
(Note that the same statistical test--Pearson's $r$--is used in both cases.)
I understand there isn't a consensus on this matter (as with many other matters in statistics as I am learning fast!) but some broad guidelines would be most helpful.
 A: I'd say that it depends on your research questions. Are the two research questions related to the same data set? Are they completely independent from eachother? Would a false negative be worse than a false positive for either of these questions? If you have jellybeans type questions, correct for all of them. If your first question is about the link between smoking and cancer and your second question is about the link between a father's length and his son's length, correct for 4 in each batch. If you're somewhere in between, I don't think that there is a "right answer" to this.
You could correct for all 8 tests. But then again, you could correct for all 400 tests that you run this year, to get at most a 5 % (say) risk of making a type I error in 2012. You have to draw the line somewhere...
Also, as you run multiple correlation tests, it sounds like you might have multivariate data. Have you considered using some multivariate generalization of Pearson's correlation coefficient, such as canonical correlations?
