Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

When we perform experiments (on small sample sizes (usually the sample size per treatment group is about 7~8)) on two groups, we use a t-test to test for difference. However, when we perform an ANOVA (obviously for more than two groups), we use something along the lines of Bonferroni (LSD/# of pairwise comparisons) or Tukey's as a post hoc, and as a student, I have been warned off from using Fisher's Least Significant Difference (LSD).

Now the thing is, LSD is similar to pairwise t-test (am I right?), and so the only thing it doesn't account for is that we're doing multiple comparisons. How important is that when dealing with say 6 groups, if the ANOVA is itself significant?

Or in other words, is there any scientific/statistical reason for using a Fisher's LSD?

share|improve this question
1  
Could you clarify who 'they' are and include examples of what they say? (Just how bad do they say it is? Let's be clear what we're discussing here.) I've seen some criticism of LSD, some of it justified, but I don't know that I've seen what you've seen, nor that what I've seen would require quite the characterization you have there. –  Glen_b Jul 10 at 0:46
    
+1 if you came here trying to figure out which stack exchange site talks about Timothy Learys LSD –  PW Kad Jul 10 at 13:59
    
@Glen_b They refers to scientists in Biomedical sciences. The words of my professor were, to quote "Use Bonferroni or Tukey. Use LSD only in desperation. If that doesn't help, use the other LSD :) " –  Rover Eye Jul 10 at 15:56

3 Answers 3

up vote 5 down vote accepted

Fisher's LSD is indeed a series of pairwise t-tests, with each test using the mean squared error from the significant ANOVA as its pooled variance estimate (and naturally taking the associated degrees of freedom). That the ANOVA be significant is an additional constraint of this test.

It restricts family-wise error rate to alpha in the special case of 3 groups only. Howell has a very good and relatively simple explanation of how it does so in Chapter 16 of his book Fundamental Statistics for the Behavioral Sciences, 8th edition, David C. Howell.

Above 3 groups alpha inflates rapidly (as @Alexis has noted above). It is not certainly appropriate for 6 groups. I believe that it is this limited applicability that causes most people to suggest ignoring it as an option.

share|improve this answer

How important are multiple comparisons when dealing with 6 groups? Well... with six groups you are dealing with a maximum of $\frac{6(6-1)}{2} = 15$ possible post hoc pairwise comparisons. I will let the inestimable Randall Munroe address the importance of multiple comparisons:

enter image description here

And I will add that if, as in your opening sentence, you suggest that sometimes you have seven groups, then the maximum number of post hoc pairwise tests is $\frac{7(7-1)}{2} = 21$, which is far too similar to the jellybean scenario just presented (which also presents 21 tests ;). So, really, unless you want to world to mock you by repeatedly sending you copies of xkcd 882, I would just go ahead and perform multiple comparisons adjustments (either FWER, like the Bonferroni or Holm-Sidak, or FDR like Benjamini and Hochberg).

share|improve this answer
    
Point well made. However, doesn't this bring to question, that if we have very few groups (say 3 (3 pairwise) or 4 (6 pairwise)) the probability of finding a significant value by chance is low ? (again, the LSD is protected by ANOVA's significance) –  Rover Eye Jul 9 at 18:08
2  
I am not sure I understand what you are asking after. If you conduct more than a single hypothesis test, then the substantive meaning of $\alpha$ and its relationship to Type I errors no longer obtains, because it is explicitly applied to a single test (hence the need for FWER or FDR). If you do not care about Type I error rates, then why conduct hypothesis tests at all? –  Alexis Jul 9 at 18:10
2  
@Rover Having 6 pairwise tests that all pass with p > 0.05 already means you now have p > 0.26, that's a rather significant change. –  Voo Jul 10 at 0:46
    
I am not asking after anything practical, was just musing out loud. But your point is well made. @Voo true, the error tends to multiply. Thanks you both. –  Rover Eye Jul 10 at 16:06

Fisher's test is as bad as everyone says it is from a Neyman-Pearson point of view and if you do what your question implies---after a significant ANOVA test each individual difference. You can see this in many published papers. But, testing all the differences after an ANOVA, or any of them, is neither necessary nor recommended. And, Fisher's test wasn't crafted under a Neyman-Pearson theory of statistical inference.

It is important to keep in mind that, when Fisher proposed the LSD, he didn't really consider multiple testing an important problem because he didn't consider the significance cutoff a hard and fast rule for deciding whether results were important or not. One could construct an LSD as an easy way to peruse the data for where there might be significant results but not the arbiter of what was meaningful. Remember, it was Fisher who said that you should just run more subjects if p > 0.05.

And why would you think that testing everything is a good idea? Consider why you run an ANOVA in the first place. You were probably taught that it's because running multiple t-tests is problematic, as you intimate in your question. Then why are you running them, or their equivalent afterward? I know it happens but I have yet to ever need to run a test after an ANOVA. An ANOVA tells you that your pattern of data is not a set of equal values, that there may be some meaning in there. Many people get hung up on the caution that the test does not tell you where the meaningful bits are but they forget that the data, and theories, tell you that.

share|improve this answer
    
Thanks for the papers. you raise a question as to why do people use a post-hoc after an ANOVA. To tell you the honest truth, I really don't know. I was told that the ANOVA is a blob test and as you mentioned, we needed to find out where the significance lies. And to be honest, I am interested in knowing how you report an ANOVA only. –  Rover Eye Jul 10 at 16:01
    
Give me a data pattern and I'll tell you how I'd report the ANOVA. The short version is that you describe the data. Items very close to each other would be grouped in the description and ones far apart considered meaningful differences (but it's all relative). Let's say I have A = 20, B = 58, C = 61, p = 0.03. I'd report the statistic and say that A is lower than B and C, which are similar. So, it all depends on the data. I can imagine a sequence of items being a bit troublesome, (A = 10, B = 20, C = 30) in some inferential ways but then perhaps I should have done a regression. –  John Jul 10 at 16:10
    
That's quite an interesting way to report an ANOVA and I can see what you are getting at. While I can surely discuss this with my supervisors, I am not too sure as to if they'll like to break the "norm" of reporting an ANOVA without a posthoc. Trying to find scientific publications that have reported using this method. –  Rover Eye Jul 10 at 16:18
1  
Something is meaningful in the data. Quite often it's obvious what that must be. Doing a post hoc to demonstrate the obvious just demonstrates you don't know what the ANOVA does in the first place. –  John Jul 11 at 1:46

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.