Post-hocs for within subjects tests?

What is the preferred method for for conducting post-hocs for within subjects tests? I've seen published work where Tukey's HSD is employed but a review of Keppel and Maxwell & Delaney suggests that the likely violation of sphericity in these designs makes the error term incorrect and this approach problematic. Maxwell & Delaney provide an approach to the problem in their book, but I've never seen it done that way in any stats package. Is the approach they offer appropriate? Would a Bonferroni or Sidak correction on multiple paired sample t-tests be reasonable? An acceptable answer will provide general R code which can conduct post-hocs on simple, multiple-way, and mixed designs as produced by the ezANOVA function in the ez package, and appropriate citations that are likely to pass muster with reviewers.

• This article by David Howell explains the problems and several solutions. – Harvey Motulsky Jul 27 '10 at 5:23
• as you accepted the answer using the multcomp package, could you elaborate a little on how you finally used multcomp. Are you using it with the lmeor lmer function or with some more traditional methods as t-test or ANOVA (as I am currently trying to use it with ANOVAs). – Henrik Aug 15 '11 at 10:19
• I accepted the multcomp answer primarily because I'm completely unsatisfied with the p-value adjustment techniques which the community selected as the "right" answer. I glanced at it and it seemed promising, but I didn't investigate further. I'd be interested in hearing more about what you are trying and what you are finding out. – russellpierce Aug 16 '11 at 1:56
• I found a way of specifying a repeated-measures ANOVA using lme, see the comments to the accepted answer: stats.stackexchange.com/q/14088/442 With an object of class lme you can use multcomp for within-subject effects. It offers different types of alpha-error adjustment, but mostly those you not especially like (as the one I proposed that was voted the "right" by the community). Besides the vignette, there is also a book on multcomp that explains all methods. If you want post-hocs without adjustment, use either fit.contrast from gmodel or the new contrast package. – Henrik Aug 31 '11 at 20:53
• Are you still interested in a solution for the ezANOVA function? If so, I think I can answer that Q but the A would rely on tests for univariate models for which sphericity is a critical assumption. If you don't need the A to be constrained to the ANOVA calculations of the ez package, I could give an A that uses multivariate models for the post-hoc tests. – statmerkur Jan 24 at 8:34

Have a look at the multcomp-package and its vignette Simultaneous Inference in General Parametric Models. I think it should do what wan't and the vignette has very good examples and extensive references.

I am currently writing a paper in which I have the pleasure to conduct both between and within subjects comparisons. After discussion with my supervisor we decided to run t-tests and use the pretty simple Holm-Bonferroni method (wikipedia) for correcting for alpha error cumulation. It controls for familwise error rate but has a greater power than the ordinary Bonferroni procedure. Procedure:

1. You run the t-tests for all comparisons you want to do.
2. You order the p-values according to their value.
3. You test the smallest p-value against alpha / k, the second smallest against alpha /( k - 1), and so forth until the first test turns out non-significant in this sequence of tests.

Cite Holm (1979) which can be downloaded via the link at wikipedia.

• maybe an ANOVA before multiple tests ? – stan Sep 26 '11 at 3:52
• I think that was implied by the answer. You perform the post-hoc tests after the signifcant ANOVA. – Henrik Sep 26 '11 at 9:59
• @Henrik: I hope I am not beating a dead horse here...by posting on an old post. So I have a question about the way you ran the t-tests. Did you use the pooled variance (from the ANOVA) or did you simply do independent pairwise t-tests? The reason I ask this is because I tried to use pairwise.t.test() to do pairwise comparisons using either the Bonferroni method or the Holm-Bonf method, but the results differ drastically depending on whether I use the pooled sd or treat each comparison as a separate, independent t-test. Thanks! – Alex Feb 27 '13 at 14:55
• @Alex: Using a 'protected' approach where the t-tests are only performed after a significant ANOVA implies the use of the pooled error term. However, because this isn't an option frequently provided by statistical software, people tend not to do it. Moreover, to the extent sphericity is violated it is a questionable thing to do in the first place. – russellpierce Jul 18 '13 at 10:10

I recall some discussion on this in the past; I'm not aware of any implementation of Maxwell & Delaney's approach, although it shouldn't be too difficult to do. Have a look at "Repeated Measures ANOVA using R" which also shows one method of addressing the sphericity issue in Tukey's HSD.

You might also find this description of Friedman's test of interest.

• Thanks, I think the Friedman test is interesting, but I can't quite figure out how it is doing that adjustment for Type I error in the post-hoc. The comments say it is a "Wilcoxon-Nemenyi-McDonald-Thompson test" but I've never heard of that before could you explain it? – russellpierce Jul 25 '10 at 4:29
• @Shane The first link is dead :-( – Adam Ryczkowski Jun 14 '16 at 13:37

There are TWO options for the inferential F-tests In SPSS. Multivariate does NOT assume sphericity, adn so makes use of a different pairwise correlation for each pair of variables. The "tests of within subjects effects", including any post hoc tests, assumes sphericity and makes some corrections for using a common correlation across all tests. These procedures are a legacy of the days when computation was expensive, and are a waste of time with modern computing facilities.

My recommendation is to take the omnibus MULTIVARIATE F for any repeated measures. Then follow up with post hoc pairwise t-test, or ANOVA with only 2 levels in each repeated measure comparison if there are also between subject factors. I would make the simple bon ferroni correction of dividing the alpha level by the number of tests.

Also be sure to look at the effect size [available in the option dialogue]. Large effect sizes that are 'close' to significant may be more worthy of attention [and future experiments] than small, but significant effects.

A more sophisticated approach is available in SPSS procedure MIXED, and also in less user friendly [but free] packages such as R.

Summary, in SPSSS, multivariate F followed by pairwise post hocs eith Bon Ferroniwith Bonferroni should be sufficient for most needs.

I shall use R function qtukey(1-alpha, means, df) to make family-wise CIs.

For example, R function qtukey(1-0.05, nmeans=4, df=16) gave the critical value $$tukey_{0.05,4,16}$$=4.046093.

Given a between-subject design with k=4 groups, 5*k=20 sample size e.g. (5-1)*k=16 df for $$MS_{Error}$$, \begin{align} & Tuke{{y}_{k,df}}=\frac{Ma{{x}_{j=1,2,\ldots ,k}}\left\{ {{z}_{j}} \right\}-Mi{{n}_{j=1,2,\ldots ,k}}\left\{ {{z}_{j}} \right\}}{\sqrt{\chi _{df}^{2}/df}} \\ & =\frac{Rang{{e}_{j=1,2,\ldots ,k}}\left\{ \frac{{{M}_{j}}-{{\mu }_{j}}}{{{\sigma }_{M}}} \right\}}{S{{E}_{M}}/{{\sigma }_{M}}} \\ & =\frac{Rang{{e}_{j=1,2,\ldots ,k}}\left\{ {{M}_{j}}-{{\mu }_{j}} \right\}}{S{{E}_{M}}} \\ & =\frac{Ma{{x}_{1\le {{j}_{1}},{{j}_{2}}\le k}}\left\{ \left| \left( {{M}_{{{j}_{1}}}}-{{\mu }_{{{j}_{1}}}} \right)-\left( {{M}_{{{j}_{2}}}}-{{\mu }_{{{j}_{2}}}} \right) \right| \right\}}{S{{E}_{M}}} \\ & =\frac{Ma{{x}_{1\le {{j}_{1}},{{j}_{2}}\le k}}\left\{ \left| \left( {{M}_{{{j}_{1}}}}-{{M}_{{{j}_{2}}}} \right)-\left( {{\mu }_{{{j}_{1}}}}-{{\mu }_{{{j}_{2}}}} \right) \right| \right\}}{S{{E}_{M}}} \\ \end{align}

The radius of family-wise 1-α CIs is $$S{{E}_{M}}\times tuke{{y}_{\alpha ,4,16}}=\sqrt{\frac{M{{S}_{Error}}}{5}}\times tuke{{y}_{\alpha ,4,16}}$$ because-- \begin{align} & \left\{ Tuke{{y}_{k,df}}\le tuke{{y}_{0.05,4,16}} \right\} \\ & =\left\{ \frac{Ma{{x}_{1\le {{j}_{1}},{{j}_{2}}\le k}}\left\{ \left| \left( {{M}_{{{j}_{1}}}}-{{M}_{{{j}_{2}}}} \right)-\left( {{\mu }_{{{j}_{1}}}}-{{\mu }_{{{j}_{2}}}} \right) \right| \right\}}{S{{E}_{M}}}\le tuke{{y}_{.05,4,16}} \right\} \\ & ={{\cap }_{1\le {{j}_{1}},{{j}_{2}}\le k}}\left\{ \left| \left( {{M}_{{{j}_{1}}}}-{{M}_{{{j}_{2}}}} \right)-\left( {{\mu }_{{{j}_{1}}}}-{{\mu }_{{{j}_{2}}}} \right) \right|\le S{{E}_{M}}\times tuke{{y}_{.05,4,16}} \right\} \\ \end{align}

Given a within-subject design with k=4 levels, 17 sample size e.g. (17-1)=16 df for $$MS_{Error}$$, and $${{X}_{i,j}}=\left( {{\mu }_{j}}+{{v}_{i}} \right)+{{\varepsilon }_{i,j}}={{\widetilde{X}}_{i,j}}+{{\varepsilon }_{i,j}}$$, the radius of family-wise (1-α) CIs is $$\sqrt{M{{S}_{Error}}/17}\times tuke{{y}_{\alpha ,4,16}}$$ because--

\begin{align} & Tuke{{y}_{k,df}}=\frac{Ma{{x}_{j=1,2,\ldots ,k}}\left\{ {{z}_{j}} \right\}-Mi{{n}_{j=1,2,\ldots ,k}}\left\{ {{z}_{j}} \right\}}{\sqrt{\chi _{df}^{2}/df}} \\ & =\frac{Rang{{e}_{j=1,2,\ldots ,k}}\left\{ \frac{Mea{{n}_{1\le i\le n}}\left\{ {{\widetilde{X}}_{i,j}}+{{\varepsilon }_{i,j}} \right\}-Mea{{n}_{1\le i\le n}}\left\{ {{\widetilde{X}}_{i,j}} \right\}}{{{\sigma }_{Mea{{n}_{1\le i\le n}}\left\{ {{\varepsilon }_{i,j}} \right\}}}} \right\}}{{{{\hat{\sigma }}}_{Mea{{n}_{1\le i\le n}}\left\{ {{\varepsilon }_{i,j}} \right\}}}/{{\sigma }_{Mea{{n}_{1\le i\le n}}\left\{ {{\varepsilon }_{i,j}} \right\}}}} \\ & =\frac{Rang{{e}_{j=1,2,\ldots ,k}}\left\{ {{M}_{j}}-\left( {{\mu }_{j}}+Mea{{n}_{1\le i\le n}}\left\{ {{v}_{i}} \right\} \right) \right\}}{{{{\hat{\sigma }}}_{Mea{{n}_{1\le i\le n}}\left\{ {{\varepsilon }_{i,j}} \right\}}}} \\ & =\frac{Rang{{e}_{j=1,2,\ldots ,k}}\left\{ {{M}_{j}}-{{\mu }_{j}} \right\}}{\sqrt{M{{S}_{Error}}/n}} \\ & =\frac{Ma{{x}_{1\le {{j}_{1}},{{j}_{2}}\le k}}\left\{ \left| \left( {{M}_{{{j}_{1}}}}-{{M}_{{{j}_{2}}}} \right)-\left( {{\mu }_{{{j}_{1}}}}-{{\mu }_{{{j}_{2}}}} \right) \right| \right\}}{\sqrt{M{{S}_{Error}}/n}} \\ \end{align}