8
$\begingroup$

I have been asked to use protected paired samples t-tests in an analysis. The requester states that if I do not use the overall MSe from my (1 factor with four levels) within-subjects ANOVA when conducting my paired samples t-tests there is not really any protection from the ANOVA.

As I remember it, in a between subjects ANOVA this procedure is only defensible if the homogeneity of variance assumption is met. It seems like a likely extension to within-subjects ANOVA might be that this is only permissible if there is no violation of sphericity. Because there are violations in this dataset, I have chosen to apply the Huynh-Feldt correction for sphericity. Regardless, if anything such an approach seems anti-conservative as it provides more degrees of freedom in the denominator. In addition, the help file in R for pairwise.t.test says that "pooling does not generalize to paired tests".

The purpose for my planned comparison t-tests is merely to identify the differences between conditions that have resulted in a significant ANOVA. I would like to be able to justify my reasons for rejecting the pooling of error variances, but I am unable to find a citation that clearly states that such an approach is inappropriate. Does anyone know of one? Alternatively, why is my thinking on this issue incorrect?

$\endgroup$
2
  • $\begingroup$ The answer to this question depends on the purpose for the request. Can you add that to your question? Specifically, how would the tests be used in terms of conclusions drawn? Also, what have you done to assess sphericity? What is the design of the study? $\endgroup$
    – John
    Sep 19, 2011 at 19:25
  • $\begingroup$ John, I have edited the question. I hope I've addressed all of your questions. $\endgroup$ Sep 19, 2011 at 20:08

1 Answer 1

6
$\begingroup$

I don't know of any paper that makes that explicit statement probably because it's not entirely true by itself.

You are correct that sphericity should be met. But, you've left the issue of sphericity vague in your question because "met" is ill defined and somewhat subjective. With only 4 levels you probably aren't having very large sphericity violations. Masson & Loftus (2003; Loftus & Masson, 1994) have mentioned that you should adhere to sphericity before using pooled measures in similar situations to what you describe and have given guidelines; but there's no hard and fast rule. The kinds of comparisons they're doing in those papers are equivalent to repeated measures t-tests in terms of power and error rates so you should look at them.

Then there's the whole issue of whether there's any protection from a significant ANOVA in "protected" tests. What's being requested is pretty equivalent to Fisher's protected least significant difference (PLSD). These protected tests have been demonstrated not to be protected against alpha inflation in general. A simple simulation of a 3-level ANOVA with A1<A2 and A2=A3 will show a higher likelihood to find A2,A3 differences than expected from alpha using PLSD. (reference escapes me... but not the answer you want anyway)

That said, your argument about individual variances is problematic because, even if homogeneity or sphericity are not perfect, you often get a more accurate estimate from the pooled value. Therefore, even though the whole idea of the significant F protecting the alpha is questionable, you should probably be using the pooled variance. You haven't presented any argument that you get more protection from alpha inflation using individual tests.

And with all that said...

I'm not sure what you're trying to defend, a difference you found or one you did not. Regardless, don't. If pooling the variance makes a new difference appear or something go away report that. Report your effect sizes, your beliefs about the fact that sphericity isn't met... just tell the whole story. You should also make a statement about the power you have. There's no firm ground here, in what you've presented, to argue that the reviewer is wrong in the general case.

$\endgroup$
3
  • $\begingroup$ Aren't the larger degrees of freedom present in the "protected" t-tests going to by their nature be anti-conservative? I guess it will be a subjective issue whether departures from sphericity will be sufficient to reject the pooling of variances. I guess it is also open to decide whether to use those pooled variances while doing other post-hoc procedures. Honestly, I don't care to defend a difference I found or one I did not. I'd like to accurately present the data, but short report journal articles don't always give enough space to tell the whole story. $\endgroup$ Sep 20, 2011 at 19:41
  • $\begingroup$ That the R help file stated (without citation) "pooling does not generalize to paired tests" gave me some inkling it was a fishy approach even if the assumption of sphericity was met; but, what I am extracting from your answer is that it is a reasonable approach - which is 1/2 my question. $\endgroup$ Sep 20, 2011 at 19:42
  • $\begingroup$ That's a statement about the capabilities of the function, not about whether it's appropriate or not. Thus, no citation. It would be harder to write it so that it pooled paired but trivial for unpaired. $\endgroup$
    – John
    Sep 22, 2011 at 2:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.