Dependent tests? I have 3 independent data groups: A, B, C. I performed t-test on each pair of groups: AB, AC and BC. So these three tests (AB, AC, BC) are not independent, are they? Thanks for answer!
 A: Jim Lewis is correct that the tests described in the question are not independent. However, I see several problems with Lewis' advice on how to handle the problem of multiple comparisons.
For example, it is not correct that the error rate per family is the same as the familywise error rate (Frane, 2015). Moreover, the Benjamini-Hochberg procedure (which Lewis recommended) doesn't control either of those error rates--it controls a less stringent rate that wasn't mentioned, called the false discovery rate (Benjamini & Hochberg, 1995). False discovery rate control is typically reserved for large numbers of hypotheses--not for 3 tests as in the question. Also, like the other responses, Lewis' provides a general recommendation (in this case, either not adjusting at all or using the Benjamini-Hochberg procedure) without knowing the necessary information about the situation (e.g., in a clinical trial, neither of those recommended approaches would be acceptable).
Lewis also appears to endorse the do-nothing approach of Perneger, which has been heavily criticized (e.g., see here, here, and here). Lewis' explanation for this endorsement was: "Unless, for your situation, the cost of a Type I error is much greater than the cost of a Type II error, you should avoid applying any of the techniques designed to suppress alpha inflation, including Bonferroni or Benjamini–Hochberg adjustments." That doesn't seem to make sense. Type I errors are typically considered much more costly than Type II errors in null hypothesis testing--that's why we test at alpha = .05. If Type II errors were just as bad as Type I errors, wouldn't we be testing at alpha = .5? The whole purpose of null hypothesis testing in the first place is to prevent Type I errors.
