# Is Fisher's LSD as bad as they say it is?

When we perform experiments (on small sample sizes (usually the sample size per treatment group is about 7~8)) on two groups, we use a t-test to test for difference. However, when we perform an ANOVA (obviously for more than two groups), we use something along the lines of Bonferroni (LSD/# of pairwise comparisons) or Tukey's as a post hoc, and as a student, I have been warned off from using Fisher's Least Significant Difference (LSD).

Now the thing is, LSD is similar to pairwise t-test (am I right?), and so the only thing it doesn't account for is that we're doing multiple comparisons. How important is that when dealing with say 6 groups, if the ANOVA is itself significant?

Or in other words, is there any scientific/statistical reason for using a Fisher's LSD?

-
Could you clarify who 'they' are and include examples of what they say? (Just how bad do they say it is? Let's be clear what we're discussing here.) I've seen some criticism of LSD, some of it justified, but I don't know that I've seen what you've seen, nor that what I've seen would require quite the characterization you have there. –  Glen_b Jul 10 at 0:46
+1 if you came here trying to figure out which stack exchange site talks about Timothy Learys LSD –  PW Kad Jul 10 at 13:59
@Glen_b They refers to scientists in Biomedical sciences. The words of my professor were, to quote "Use Bonferroni or Tukey. Use LSD only in desperation. If that doesn't help, use the other LSD :) " –  Rover Eye Jul 10 at 15:56

Fisher's LSD is indeed a series of pairwise t-tests, with each test using the mean squared error from the significant Anova as its pooled variance estimate (and naturally taking the associated degrees of freedom). That the Anova be significant is an additional constraint of this test.

It restricts family-wise error rate to alpha in the special case of 3 groups only. Howell has a very good and relatively simple explanation of how it does so in Chapter 16 of his book Fundamental Statistics for the Behavioral Sciences, 8th edition, David C. Howell.

Above 3 groups alpha inflates rapidly (as @Alexis has noted above). It is not certainly appropriate for 6 groups. I believe that it is this limited applicability that causes most people to suggest ignoring it as an option.

-

How important are multiple comparisons when dealing with 6 groups? Well... with six groups you are dealing with a maximum of $\frac{6(6-1)}{2} = 15$ possible post hoc pairwise comparisons. I will let the inestimable Randall Munroe address the importance of multiple comparisons:

And I will add that if, as in your opening sentence, you suggest that sometimes you have seven groups, then the maximum number of post hoc pairwise tests is $\frac{7(7-1)}{2} = 21$, which is far too similar to the jellybean scenario just presented (which also presents 21 tests ;). So, really, unless you want to world to mock you by repeatedly sending you copies of xkcd 882, I would just go ahead and perform multiple comparisons adjustments (either FWER, like the Bonferroni or Holm-Sidak, or FDR like Benjamini and Hochberg).

-
Point well made. However, doesn't this bring to question, that if we have very few groups (say 3 (3 pairwise) or 4 (6 pairwise)) the probability of finding a significant value by chance is low ? (again, the LSD is protected by ANOVA's significance) –  Rover Eye Jul 9 at 18:08
I am not sure I understand what you are asking after. If you conduct more than a single hypothesis test, then the substantive meaning of $\alpha$ and its relationship to Type I errors no longer obtains, because it is explicitly applied to a single test (hence the need for FWER or FDR). If you do not care about Type I error rates, then why conduct hypothesis tests at all? –  Alexis Jul 9 at 18:10
@Rover Having 6 pairwise tests that all pass with p > 0.05 already means you now have p > 0.26, that's a rather significant change. –  Voo Jul 10 at 0:46
I am not asking after anything practical, was just musing out loud. But your point is well made. @Voo true, the error tends to multiply. Thanks you both. –  Rover Eye Jul 10 at 16:06