When we perform experiments (on small sample sizes (usually the sample size per treatment group is about 7~8)) on two groups, we use a t-test to test for difference. However, when we perform an ANOVA (obviously for more than two groups), we use something along the lines of Bonferroni (LSD/# of pairwise comparisons) or Tukey's as a post hoc, and as a student, I have been warned off from using Fisher's Least Significant Difference (LSD).

Now the thing is, LSD is similar to pairwise t-test (am I right?), and so the only thing it doesn't account for is that we're doing multiple comparisons. How important is that when dealing with say 6 groups, if the ANOVA is itself significant?

Or in other words, is there any scientific/statistical reason for using a Fisher's LSD?

  • 2
    $\begingroup$ Could you clarify who 'they' are and include examples of what they say? (Just how bad do they say it is? Let's be clear what we're discussing here.) I've seen some criticism of LSD, some of it justified, but I don't know that I've seen what you've seen, nor that what I've seen would require quite the characterization you have there. $\endgroup$ – Glen_b Jul 10 '14 at 0:46
  • $\begingroup$ +1 if you came here trying to figure out which stack exchange site talks about Timothy Learys LSD $\endgroup$ – PW Kad Jul 10 '14 at 13:59
  • $\begingroup$ @Glen_b They refers to scientists in Biomedical sciences. The words of my professor were, to quote "Use Bonferroni or Tukey. Use LSD only in desperation. If that doesn't help, use the other LSD :) " $\endgroup$ – Rover Eye Jul 10 '14 at 15:56

Fisher's LSD is indeed a series of pairwise t-tests, with each test using the mean squared error from the significant ANOVA as its pooled variance estimate (and naturally taking the associated degrees of freedom). That the ANOVA be significant is an additional constraint of this test.

It restricts family-wise error rate to alpha in the special case of 3 groups only. Howell has a very good and relatively simple explanation of how it does so in Chapter 16 of his book Fundamental Statistics for the Behavioral Sciences, 8th edition, David C. Howell.

Above 3 groups alpha inflates rapidly (as @Alexis has noted above). It is not certainly appropriate for 6 groups. I believe that it is this limited applicability that causes most people to suggest ignoring it as an option.


How important are multiple comparisons when dealing with 6 groups? Well... with six groups you are dealing with a maximum of $\frac{6(6-1)}{2} = 15$ possible post hoc pairwise comparisons. I will let the inestimable Randall Munroe address the importance of multiple comparisons:

enter image description here

And I will add that if, as in your opening sentence, you suggest that sometimes you have seven groups, then the maximum number of post hoc pairwise tests is $\frac{7(7-1)}{2} = 21$, which is far too similar to the jellybean scenario just presented (which also presents 21 tests ;). So, really, unless you want to world to mock you by repeatedly sending you copies of xkcd 882, I would just go ahead and perform multiple comparisons adjustments (either FWER, like the Bonferroni or Holm-Sidak, or FDR like Benjamini and Hochberg).

  • 1
    $\begingroup$ Point well made. However, doesn't this bring to question, that if we have very few groups (say 3 (3 pairwise) or 4 (6 pairwise)) the probability of finding a significant value by chance is low ? (again, the LSD is protected by ANOVA's significance) $\endgroup$ – Rover Eye Jul 9 '14 at 18:08
  • 2
    $\begingroup$ I am not sure I understand what you are asking after. If you conduct more than a single hypothesis test, then the substantive meaning of $\alpha$ and its relationship to Type I errors no longer obtains, because it is explicitly applied to a single test (hence the need for FWER or FDR). If you do not care about Type I error rates, then why conduct hypothesis tests at all? $\endgroup$ – Alexis Jul 9 '14 at 18:10
  • 2
    $\begingroup$ @Rover Having 6 pairwise tests that all pass with p > 0.05 already means you now have p > 0.26, that's a rather significant change. $\endgroup$ – Voo Jul 10 '14 at 0:46
  • $\begingroup$ I am not asking after anything practical, was just musing out loud. But your point is well made. @Voo true, the error tends to multiply. Thanks you both. $\endgroup$ – Rover Eye Jul 10 '14 at 16:06

Fisher's test is as bad as everyone says it is from a Neyman-Pearson point of view and if you do what your question implies---after a significant ANOVA test each individual difference. You can see this in many published papers. But, testing all the differences after an ANOVA, or any of them, is neither necessary nor recommended. And, Fisher's test wasn't crafted under a Neyman-Pearson theory of statistical inference.

It is important to keep in mind that, when Fisher proposed the LSD, he didn't really consider multiple testing an important problem because he didn't consider the significance cutoff a hard and fast rule for deciding whether results were important or not. One could construct an LSD as an easy way to peruse the data for where there might be significant results but not the arbiter of what was meaningful. Remember, it was Fisher who said that you should just run more subjects if p > 0.05.

And why would you think that testing everything is a good idea? Consider why you run an ANOVA in the first place. You were probably taught that it's because running multiple t-tests is problematic, as you intimate in your question. Then why are you running them, or their equivalent afterward? I know it happens but I have yet to ever need to run a test after an ANOVA. An ANOVA tells you that your pattern of data is not a set of equal values, that there may be some meaning in there. Many people get hung up on the caution that the test does not tell you where the meaningful bits are but they forget that the data, and theories, tell you that.

  • $\begingroup$ Thanks for the papers. you raise a question as to why do people use a post-hoc after an ANOVA. To tell you the honest truth, I really don't know. I was told that the ANOVA is a blob test and as you mentioned, we needed to find out where the significance lies. And to be honest, I am interested in knowing how you report an ANOVA only. $\endgroup$ – Rover Eye Jul 10 '14 at 16:01
  • 1
    $\begingroup$ Give me a data pattern and I'll tell you how I'd report the ANOVA. The short version is that you describe the data. Items very close to each other would be grouped in the description and ones far apart considered meaningful differences (but it's all relative). Let's say I have A = 20, B = 58, C = 61, p = 0.03. I'd report the statistic and say that A is lower than B and C, which are similar. So, it all depends on the data. I can imagine a sequence of items being a bit troublesome, (A = 10, B = 20, C = 30) in some inferential ways but then perhaps I should have done a regression. $\endgroup$ – John Jul 10 '14 at 16:10
  • $\begingroup$ That's quite an interesting way to report an ANOVA and I can see what you are getting at. While I can surely discuss this with my supervisors, I am not too sure as to if they'll like to break the "norm" of reporting an ANOVA without a posthoc. Trying to find scientific publications that have reported using this method. $\endgroup$ – Rover Eye Jul 10 '14 at 16:18
  • 2
    $\begingroup$ Something is meaningful in the data. Quite often it's obvious what that must be. Doing a post hoc to demonstrate the obvious just demonstrates you don't know what the ANOVA does in the first place. $\endgroup$ – John Jul 11 '14 at 1:46

The reasoning behind Fisher's LSD can be extended to cases beyond N=3.

I'll discuss the case of four groups in detail. To keep the familywise Type-I error rate at 0.05 or below, a multiple-comparison correction factor of 3 (i.e. a per-comparison alpha of 0.05/3) suffices, although there are six post-hoc comparisons among the four groups. This is because:

  • in case all four true means are equal, the omnibus Anova over the four groups limits the familywise error rate to 0.05;
  • in case three of the true means are equal and the fourth differs from them, there are only three comparisons that could potentially yield a Type-I error;
  • in case two of the true means are equal and differ from the other two, which are equal to each other, there are only two comparisons that could potentially yield a Type-I error.

This exhausts the possibilities. In all cases, the probability of finding one or more p-values below 0.05 for groups whose true means are equal, stays at or below 0.05 if the correction factor for multiple comparisons is 3, and this is the definition of the familywise error rate.

This reasoning for four groups is a generalization from Fisher's explanation for his three-group Least Significant Difference method. For N groups, the correction factor, if the omnibus Anova test is significant, is (N-1)(N-2)/2. So the Bonferroni correction, by a factor of N(N-1)/2, is too strong. It suffices to use an alpha correction factor of 1 for N=3 (this is why Fisher's LSD works for N=3), a factor of 3 for N=4, a factor of 6 for N=5, a factor of 10 for N=6, and so on.

  • $\begingroup$ +1. This is a very good addition to the thread. Welcome to the site! $\endgroup$ – amoeba Feb 12 '16 at 10:16
  • $\begingroup$ Every situation you described does not required any post hoc testing. $\endgroup$ – John Feb 12 '16 at 14:38
  • 1
    $\begingroup$ Can you point to a literature reference for that $(N-1)(N-2)/2$ result? $\endgroup$ – Russ Lenth Feb 12 '16 at 16:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.