Post-hocs for within subjects tests?

Question

What is the preferred method for for conducting post-hocs for within subjects tests? I've seen published work where Tukey's HSD is employed but a review of Keppel and Maxwell & Delaney suggests that the likely violation of sphericity in these designs makes the error term incorrect and this approach problematic. Maxwell & Delaney provide an approach to the problem in their book, but I've never seen it done that way in any stats package. Is the approach they offer appropriate? Would a Bonferroni or Sidak correction on multiple paired sample t-tests be reasonable? An acceptable answer will provide general R code which can conduct post-hocs on simple, multiple-way, and mixed designs as produced by the ezANOVA function in the ez package, and appropriate citations that are likely to pass muster with reviewers.

Answer 1

Have a look at the multcomp-package and its vignette Simultaneous Inference in General Parametric Models. I think it should do what wan't and the vignette has very good examples and extensive references.

Answer 2

I am currently writing a paper in which I have the pleasure to conduct both between and within subjects comparisons. After discussion with my supervisor we decided to run t-tests and use the pretty simple Holm-Bonferroni method (wikipedia) for correcting for alpha error cumulation. It controls for familwise error rate but has a greater power than the ordinary Bonferroni procedure. Procedure:

1. You run the t-tests for all comparisons you want to do.
2. You order the p-values according to their value.
3. You test the smallest p-value against alpha / k, the second smallest against alpha /( k - 1), and so forth until the first test turns out non-significant in this sequence of tests.

Cite Holm (1979) which can be downloaded via the link at wikipedia.

Answer 3

I recall some discussion on this in the past; I'm not aware of any implementation of Maxwell & Delaney's approach, although it shouldn't be too difficult to do. Have a look at "Repeated Measures ANOVA using R" which also shows one method of addressing the sphericity issue in Tukey's HSD.

You might also find this description of Friedman's test of interest.

Answer 4

There are TWO options for the inferential F-tests In SPSS. Multivariate does NOT assume sphericity, adn so makes use of a different pairwise correlation for each pair of variables. The "tests of within subjects effects", including any post hoc tests, assumes sphericity and makes some corrections for using a common correlation across all tests. These procedures are a legacy of the days when computation was expensive, and are a waste of time with modern computing facilities.

My recommendation is to take the omnibus MULTIVARIATE F for any repeated measures. Then follow up with post hoc pairwise t-test, or ANOVA with only 2 levels in each repeated measure comparison if there are also between subject factors. I would make the simple bon ferroni correction of dividing the alpha level by the number of tests.

Also be sure to look at the effect size [available in the option dialogue]. Large effect sizes that are 'close' to significant may be more worthy of attention [and future experiments] than small, but significant effects.

A more sophisticated approach is available in SPSS procedure MIXED, and also in less user friendly [but free] packages such as R.

Summary, in SPSSS, multivariate F followed by pairwise post hocs eith Bon Ferroniwith Bonferroni should be sufficient for most needs.

Answer 5

I shall use R function qtukey(1-alpha, means, df) to make family-wise CIs.

For example, R function qtukey(1-0.05, nmeans=4, df=16) gave the critical value $tukey_{0.05,4,16}$=4.046093.

Given a between-subject design with k=4 groups, 5*k=20 sample size e.g. (5-1)*k=16 df for $MS_{Error}$, $\begin{align} & Tuke{{y}_{k,df}}=\frac{Ma{{x}_{j=1,2,\ldots ,k}}\left\{ {{z}_{j}} \right\}-Mi{{n}_{j=1,2,\ldots ,k}}\left\{ {{z}_{j}} \right\}}{\sqrt{\chi _{df}^{2}/df}} \\ & =\frac{Rang{{e}_{j=1,2,\ldots ,k}}\left\{ \frac{{{M}_{j}}-{{\mu }_{j}}}{{{\sigma }_{M}}} \right\}}{S{{E}_{M}}/{{\sigma }_{M}}} \\ & =\frac{Rang{{e}_{j=1,2,\ldots ,k}}\left\{ {{M}_{j}}-{{\mu }_{j}} \right\}}{S{{E}_{M}}} \\ & =\frac{Ma{{x}_{1\le {{j}_{1}},{{j}_{2}}\le k}}\left\{ \left| \left( {{M}_{{{j}_{1}}}}-{{\mu }_{{{j}_{1}}}} \right)-\left( {{M}_{{{j}_{2}}}}-{{\mu }_{{{j}_{2}}}} \right) \right| \right\}}{S{{E}_{M}}} \\ & =\frac{Ma{{x}_{1\le {{j}_{1}},{{j}_{2}}\le k}}\left\{ \left| \left( {{M}_{{{j}_{1}}}}-{{M}_{{{j}_{2}}}} \right)-\left( {{\mu }_{{{j}_{1}}}}-{{\mu }_{{{j}_{2}}}} \right) \right| \right\}}{S{{E}_{M}}} \\ \end{align}$

The radius of family-wise 1-α CIs is $S{{E}_{M}}\times tuke{{y}_{\alpha ,4,16}}=\sqrt{\frac{M{{S}_{Error}}}{5}}\times tuke{{y}_{\alpha ,4,16}}$ because-- $$ \begin{align} & \left\{ Tuke{{y}_{k,df}}\le tuke{{y}_{0.05,4,16}} \right\} \\ & =\left\{ \frac{Ma{{x}_{1\le {{j}_{1}},{{j}_{2}}\le k}}\left\{ \left| \left( {{M}_{{{j}_{1}}}}-{{M}_{{{j}_{2}}}} \right)-\left( {{\mu }_{{{j}_{1}}}}-{{\mu }_{{{j}_{2}}}} \right) \right| \right\}}{S{{E}_{M}}}\le tuke{{y}_{.05,4,16}} \right\} \\ & ={{\cap }_{1\le {{j}_{1}},{{j}_{2}}\le k}}\left\{ \left| \left( {{M}_{{{j}_{1}}}}-{{M}_{{{j}_{2}}}} \right)-\left( {{\mu }_{{{j}_{1}}}}-{{\mu }_{{{j}_{2}}}} \right) \right|\le S{{E}_{M}}\times tuke{{y}_{.05,4,16}} \right\} \\ \end{align} $$

Given a within-subject design with k=4 levels, 17 sample size e.g. (17-1)=16 df for $MS_{Error}$, and ${{X}_{i,j}}=\left( {{\mu }_{j}}+{{v}_{i}} \right)+{{\varepsilon }_{i,j}}={{\widetilde{X}}_{i,j}}+{{\varepsilon }_{i,j}}$, the radius of family-wise (1-α) CIs is $\sqrt{M{{S}_{Error}}/17}\times tuke{{y}_{\alpha ,4,16}}$ because--

$$\begin{align} & Tuke{{y}_{k,df}}=\frac{Ma{{x}_{j=1,2,\ldots ,k}}\left\{ {{z}_{j}} \right\}-Mi{{n}_{j=1,2,\ldots ,k}}\left\{ {{z}_{j}} \right\}}{\sqrt{\chi _{df}^{2}/df}} \\ & =\frac{Rang{{e}_{j=1,2,\ldots ,k}}\left\{ \frac{Mea{{n}_{1\le i\le n}}\left\{ {{\widetilde{X}}_{i,j}}+{{\varepsilon }_{i,j}} \right\}-Mea{{n}_{1\le i\le n}}\left\{ {{\widetilde{X}}_{i,j}} \right\}}{{{\sigma }_{Mea{{n}_{1\le i\le n}}\left\{ {{\varepsilon }_{i,j}} \right\}}}} \right\}}{{{{\hat{\sigma }}}_{Mea{{n}_{1\le i\le n}}\left\{ {{\varepsilon }_{i,j}} \right\}}}/{{\sigma }_{Mea{{n}_{1\le i\le n}}\left\{ {{\varepsilon }_{i,j}} \right\}}}} \\ & =\frac{Rang{{e}_{j=1,2,\ldots ,k}}\left\{ {{M}_{j}}-\left( {{\mu }_{j}}+Mea{{n}_{1\le i\le n}}\left\{ {{v}_{i}} \right\} \right) \right\}}{{{{\hat{\sigma }}}_{Mea{{n}_{1\le i\le n}}\left\{ {{\varepsilon }_{i,j}} \right\}}}} \\ & =\frac{Rang{{e}_{j=1,2,\ldots ,k}}\left\{ {{M}_{j}}-{{\mu }_{j}} \right\}}{\sqrt{M{{S}_{Error}}/n}} \\ & =\frac{Ma{{x}_{1\le {{j}_{1}},{{j}_{2}}\le k}}\left\{ \left| \left( {{M}_{{{j}_{1}}}}-{{M}_{{{j}_{2}}}} \right)-\left( {{\mu }_{{{j}_{1}}}}-{{\mu }_{{{j}_{2}}}} \right) \right| \right\}}{\sqrt{M{{S}_{Error}}/n}} \\ \end{align}$$