What is the preferred method for for conducting post-hocs for within subjects tests? I've seen published work where Tukey's HSD is employed but a review of Keppel and Maxwell & Delaney suggests that the likely violation of sphericity in these designs makes the error term incorrect and this approach problematic. Maxwell & Delaney provide an approach to the problem in their book, but I've never seen it done that way in any stats package. Is the approach they offer appropriate? Would a Bonferroni or Sidak correction on multiple paired sample t-tests be reasonable? An acceptable answer will provide general R code which can conduct post-hocs on simple, multiple-way, and mixed designs as produced by the ezANOVA function in the ez package, and appropriate citations that are likely to pass muster with reviewers.

  • 1
    $\begingroup$ This article by David Howell explains the problems and several solutions. $\endgroup$ Commented Jul 27, 2010 at 5:23
  • $\begingroup$ as you accepted the answer using the multcomp package, could you elaborate a little on how you finally used multcomp. Are you using it with the lmeor lmer function or with some more traditional methods as t-test or ANOVA (as I am currently trying to use it with ANOVAs). $\endgroup$
    – Henrik
    Commented Aug 15, 2011 at 10:19
  • $\begingroup$ I accepted the multcomp answer primarily because I'm completely unsatisfied with the p-value adjustment techniques which the community selected as the "right" answer. I glanced at it and it seemed promising, but I didn't investigate further. I'd be interested in hearing more about what you are trying and what you are finding out. $\endgroup$ Commented Aug 16, 2011 at 1:56
  • $\begingroup$ Are you still interested in a solution for the ezANOVA function? If so, I think I can answer that Q but the A would rely on tests for univariate models for which sphericity is a critical assumption. If you don't need the A to be constrained to the ANOVA calculations of the ez package, I could give an A that uses multivariate models for the post-hoc tests. $\endgroup$
    – statmerkur
    Commented Jan 24, 2019 at 8:34
  • $\begingroup$ @statmerkur it's been a while, but it looks to me like I was looking for posthocs that didn't need the sphericity assumption. Anything you can provide that includes compelling citations has a chance of passing muster $\endgroup$ Commented Jan 24, 2019 at 14:38

5 Answers 5


Have a look at the multcomp-package and its vignette Simultaneous Inference in General Parametric Models. I think it should do what wan't and the vignette has very good examples and extensive references.


I am currently writing a paper in which I have the pleasure to conduct both between and within subjects comparisons. After discussion with my supervisor we decided to run t-tests and use the pretty simple Holm-Bonferroni method (wikipedia) for correcting for alpha error cumulation. It controls for familwise error rate but has a greater power than the ordinary Bonferroni procedure. Procedure:

  1. You run the t-tests for all comparisons you want to do.
  2. You order the p-values according to their value.
  3. You test the smallest p-value against alpha / k, the second smallest against alpha /( k - 1), and so forth until the first test turns out non-significant in this sequence of tests.

Cite Holm (1979) which can be downloaded via the link at wikipedia.

  • 1
    $\begingroup$ maybe an ANOVA before multiple tests ? $\endgroup$
    – abc
    Commented Sep 26, 2011 at 3:52
  • 4
    $\begingroup$ I think that was implied by the answer. You perform the post-hoc tests after the signifcant ANOVA. $\endgroup$
    – Henrik
    Commented Sep 26, 2011 at 9:59
  • 4
    $\begingroup$ @Henrik: I hope I am not beating a dead horse here...by posting on an old post. So I have a question about the way you ran the t-tests. Did you use the pooled variance (from the ANOVA) or did you simply do independent pairwise t-tests? The reason I ask this is because I tried to use pairwise.t.test() to do pairwise comparisons using either the Bonferroni method or the Holm-Bonf method, but the results differ drastically depending on whether I use the pooled sd or treat each comparison as a separate, independent t-test. Thanks! $\endgroup$
    – Alex
    Commented Feb 27, 2013 at 14:55
  • 4
    $\begingroup$ @Alex: Using a 'protected' approach where the t-tests are only performed after a significant ANOVA implies the use of the pooled error term. However, because this isn't an option frequently provided by statistical software, people tend not to do it. Moreover, to the extent sphericity is violated it is a questionable thing to do in the first place. $\endgroup$ Commented Jul 18, 2013 at 10:10

I recall some discussion on this in the past; I'm not aware of any implementation of Maxwell & Delaney's approach, although it shouldn't be too difficult to do. Have a look at "Repeated Measures ANOVA using R" which also shows one method of addressing the sphericity issue in Tukey's HSD.

You might also find this description of Friedman's test of interest.

  • $\begingroup$ Thanks, I think the Friedman test is interesting, but I can't quite figure out how it is doing that adjustment for Type I error in the post-hoc. The comments say it is a "Wilcoxon-Nemenyi-McDonald-Thompson test" but I've never heard of that before could you explain it? $\endgroup$ Commented Jul 25, 2010 at 4:29
  • $\begingroup$ @Shane The first link is dead :-( $\endgroup$ Commented Jun 14, 2016 at 13:37

There are TWO options for the inferential F-tests In SPSS. Multivariate does NOT assume sphericity, adn so makes use of a different pairwise correlation for each pair of variables. The "tests of within subjects effects", including any post hoc tests, assumes sphericity and makes some corrections for using a common correlation across all tests. These procedures are a legacy of the days when computation was expensive, and are a waste of time with modern computing facilities.

My recommendation is to take the omnibus MULTIVARIATE F for any repeated measures. Then follow up with post hoc pairwise t-test, or ANOVA with only 2 levels in each repeated measure comparison if there are also between subject factors. I would make the simple bon ferroni correction of dividing the alpha level by the number of tests.

Also be sure to look at the effect size [available in the option dialogue]. Large effect sizes that are 'close' to significant may be more worthy of attention [and future experiments] than small, but significant effects.

A more sophisticated approach is available in SPSS procedure MIXED, and also in less user friendly [but free] packages such as R.

Summary, in SPSSS, multivariate F followed by pairwise post hocs eith Bon Ferroniwith Bonferroni should be sufficient for most needs.


I shall use R function qtukey(1-alpha, means, df) to make family-wise CIs.

For example, R function qtukey(1-0.05, nmeans=4, df=16) gave the critical value $tukey_{0.05,4,16}$=4.046093.

Given a between-subject design with k=4 groups, 5*k=20 sample size e.g. (5-1)*k=16 df for $MS_{Error}$, $\begin{align} & Tuke{{y}_{k,df}}=\frac{Ma{{x}_{j=1,2,\ldots ,k}}\left\{ {{z}_{j}} \right\}-Mi{{n}_{j=1,2,\ldots ,k}}\left\{ {{z}_{j}} \right\}}{\sqrt{\chi _{df}^{2}/df}} \\ & =\frac{Rang{{e}_{j=1,2,\ldots ,k}}\left\{ \frac{{{M}_{j}}-{{\mu }_{j}}}{{{\sigma }_{M}}} \right\}}{S{{E}_{M}}/{{\sigma }_{M}}} \\ & =\frac{Rang{{e}_{j=1,2,\ldots ,k}}\left\{ {{M}_{j}}-{{\mu }_{j}} \right\}}{S{{E}_{M}}} \\ & =\frac{Ma{{x}_{1\le {{j}_{1}},{{j}_{2}}\le k}}\left\{ \left| \left( {{M}_{{{j}_{1}}}}-{{\mu }_{{{j}_{1}}}} \right)-\left( {{M}_{{{j}_{2}}}}-{{\mu }_{{{j}_{2}}}} \right) \right| \right\}}{S{{E}_{M}}} \\ & =\frac{Ma{{x}_{1\le {{j}_{1}},{{j}_{2}}\le k}}\left\{ \left| \left( {{M}_{{{j}_{1}}}}-{{M}_{{{j}_{2}}}} \right)-\left( {{\mu }_{{{j}_{1}}}}-{{\mu }_{{{j}_{2}}}} \right) \right| \right\}}{S{{E}_{M}}} \\ \end{align}$

The radius of family-wise 1-α CIs is $S{{E}_{M}}\times tuke{{y}_{\alpha ,4,16}}=\sqrt{\frac{M{{S}_{Error}}}{5}}\times tuke{{y}_{\alpha ,4,16}}$ because-- $$ \begin{align} & \left\{ Tuke{{y}_{k,df}}\le tuke{{y}_{0.05,4,16}} \right\} \\ & =\left\{ \frac{Ma{{x}_{1\le {{j}_{1}},{{j}_{2}}\le k}}\left\{ \left| \left( {{M}_{{{j}_{1}}}}-{{M}_{{{j}_{2}}}} \right)-\left( {{\mu }_{{{j}_{1}}}}-{{\mu }_{{{j}_{2}}}} \right) \right| \right\}}{S{{E}_{M}}}\le tuke{{y}_{.05,4,16}} \right\} \\ & ={{\cap }_{1\le {{j}_{1}},{{j}_{2}}\le k}}\left\{ \left| \left( {{M}_{{{j}_{1}}}}-{{M}_{{{j}_{2}}}} \right)-\left( {{\mu }_{{{j}_{1}}}}-{{\mu }_{{{j}_{2}}}} \right) \right|\le S{{E}_{M}}\times tuke{{y}_{.05,4,16}} \right\} \\ \end{align} $$

Given a within-subject design with k=4 levels, 17 sample size e.g. (17-1)=16 df for $MS_{Error}$, and ${{X}_{i,j}}=\left( {{\mu }_{j}}+{{v}_{i}} \right)+{{\varepsilon }_{i,j}}={{\widetilde{X}}_{i,j}}+{{\varepsilon }_{i,j}}$, the radius of family-wise (1-α) CIs is $\sqrt{M{{S}_{Error}}/17}\times tuke{{y}_{\alpha ,4,16}}$ because--

$$\begin{align} & Tuke{{y}_{k,df}}=\frac{Ma{{x}_{j=1,2,\ldots ,k}}\left\{ {{z}_{j}} \right\}-Mi{{n}_{j=1,2,\ldots ,k}}\left\{ {{z}_{j}} \right\}}{\sqrt{\chi _{df}^{2}/df}} \\ & =\frac{Rang{{e}_{j=1,2,\ldots ,k}}\left\{ \frac{Mea{{n}_{1\le i\le n}}\left\{ {{\widetilde{X}}_{i,j}}+{{\varepsilon }_{i,j}} \right\}-Mea{{n}_{1\le i\le n}}\left\{ {{\widetilde{X}}_{i,j}} \right\}}{{{\sigma }_{Mea{{n}_{1\le i\le n}}\left\{ {{\varepsilon }_{i,j}} \right\}}}} \right\}}{{{{\hat{\sigma }}}_{Mea{{n}_{1\le i\le n}}\left\{ {{\varepsilon }_{i,j}} \right\}}}/{{\sigma }_{Mea{{n}_{1\le i\le n}}\left\{ {{\varepsilon }_{i,j}} \right\}}}} \\ & =\frac{Rang{{e}_{j=1,2,\ldots ,k}}\left\{ {{M}_{j}}-\left( {{\mu }_{j}}+Mea{{n}_{1\le i\le n}}\left\{ {{v}_{i}} \right\} \right) \right\}}{{{{\hat{\sigma }}}_{Mea{{n}_{1\le i\le n}}\left\{ {{\varepsilon }_{i,j}} \right\}}}} \\ & =\frac{Rang{{e}_{j=1,2,\ldots ,k}}\left\{ {{M}_{j}}-{{\mu }_{j}} \right\}}{\sqrt{M{{S}_{Error}}/n}} \\ & =\frac{Ma{{x}_{1\le {{j}_{1}},{{j}_{2}}\le k}}\left\{ \left| \left( {{M}_{{{j}_{1}}}}-{{M}_{{{j}_{2}}}} \right)-\left( {{\mu }_{{{j}_{1}}}}-{{\mu }_{{{j}_{2}}}} \right) \right| \right\}}{\sqrt{M{{S}_{Error}}/n}} \\ \end{align}$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.