Is Fisher's LSD as bad as they say it is?

Question

When we perform experiments (on small sample sizes (usually the sample size per treatment group is about 7~8)) on two groups, we use a t-test to test for difference. However, when we perform an ANOVA (obviously for more than two groups), we use something along the lines of Bonferroni (LSD/# of pairwise comparisons) or Tukey's as a post hoc, and as a student, I have been warned off from using Fisher's Least Significant Difference (LSD).

Now the thing is, LSD is similar to pairwise t-test (am I right?), and so the only thing it doesn't account for is that we're doing multiple comparisons. How important is that when dealing with say 6 groups, if the ANOVA is itself significant?

Or in other words, is there any scientific/statistical reason for using a Fisher's LSD?

Answer 1

Fisher's LSD is indeed a series of pairwise t-tests, with each test using the mean squared error from the significant ANOVA as its pooled variance estimate (and naturally taking the associated degrees of freedom). That the ANOVA be significant is an additional constraint of this test.

It restricts family-wise error rate to alpha in the special case of 3 groups only. Howell has a very good and relatively simple explanation of how it does so in Chapter 16 of his book Fundamental Statistics for the Behavioral Sciences, 8th edition, David C. Howell.

Above 3 groups alpha inflates rapidly (as @Alexis has noted above). It is not certainly appropriate for 6 groups. I believe that it is this limited applicability that causes most people to suggest ignoring it as an option.

Answer 2

How important are multiple comparisons when dealing with 6 groups? Well... with six groups you are dealing with a maximum of $\frac{6(6-1)}{2} = 15$ possible post hoc pairwise comparisons. I will let the inestimable Randall Munroe address the importance of multiple comparisons:

And I will add that if, as in your opening sentence, you suggest that sometimes you have seven groups, then the maximum number of post hoc pairwise tests is $\frac{7(7-1)}{2} = 21$, which is far too similar to the jellybean scenario just presented (which also presents 21 tests ;). So, really, unless you want to world to mock you by repeatedly sending you copies of xkcd 882, I would just go ahead and perform multiple comparisons adjustments (either FWER, like the Bonferroni or Holm-Sidak, or FDR like Benjamini and Hochberg).

Answer 3

Fisher's test is as bad as everyone says it is from a Neyman-Pearson point of view and if you do what your question implies---after a significant ANOVA test each individual difference. You can see this in many published papers. But, testing all the differences after an ANOVA, or any of them, is neither necessary nor recommended. And, Fisher's test wasn't crafted under a Neyman-Pearson theory of statistical inference.

It is important to keep in mind that, when Fisher proposed the LSD, he didn't really consider multiple testing an important problem because he didn't consider the significance cutoff a hard and fast rule for deciding whether results were important or not. One could construct an LSD as an easy way to peruse the data for where there might be significant results but not the arbiter of what was meaningful. Remember, it was Fisher who said that you should just run more subjects if p > 0.05.

And why would you think that testing everything is a good idea? Consider why you run an ANOVA in the first place. You were probably taught that it's because running multiple t-tests is problematic, as you intimate in your question. Then why are you running them, or their equivalent afterward? I know it happens but I have yet to ever need to run a test after an ANOVA. An ANOVA tells you that your pattern of data is not a set of equal values, that there may be some meaning in there. Many people get hung up on the caution that the test does not tell you where the meaningful bits are but they forget that the data, and theories, tell you that.

Answer 4

The reasoning behind Fisher's LSD can be extended to cases beyond N=3.

I'll discuss the case of four groups in detail. To keep the familywise Type-I error rate at 0.05 or below, a multiple-comparison correction factor of 3 (i.e. a per-comparison alpha of 0.05/3) suffices, although there are six post-hoc comparisons among the four groups. This is because:

• in case all four true means are equal, the omnibus Anova over the four groups limits the familywise error rate to 0.05;
• in case three of the true means are equal and the fourth differs from them, there are only three comparisons that could potentially yield a Type-I error;
• in case two of the true means are equal and differ from the other two, which are equal to each other, there are only two comparisons that could potentially yield a Type-I error.

This exhausts the possibilities. In all cases, the probability of finding one or more p-values below 0.05 for groups whose true means are equal, stays at or below 0.05 if the correction factor for multiple comparisons is 3, and this is the definition of the familywise error rate.

This reasoning for four groups is a generalization from Fisher's explanation for his three-group Least Significant Difference method. For N groups, the correction factor, if the omnibus Anova test is significant, is (N-1)(N-2)/2. So the Bonferroni correction, by a factor of N(N-1)/2, is too strong. It suffices to use an alpha correction factor of 1 for N=3 (this is why Fisher's LSD works for N=3), a factor of 3 for N=4, a factor of 6 for N=5, a factor of 10 for N=6, and so on.