I am examining three independent groups that were measured on a continuous outcome variable. I have a priori belief that the result should be Group 1 < Group 2 < Group 3. I've been told to do an ANOVA, and then to consider doing three t-tests to look at the specific differences between the groups.

Is there any reason to bother looking at the result of the ANOVA? It's been put to me that to keep the Type 1 error rate to 5% I should look at the ANOVA and if it's non-significant I should abandon the t-tests. However, someone else suggested to me that since t-tests could be statistically significant even when the omnibus ANOVA is not, and that I should therefore not bother looking at the ANOVA, and should do the individual t-tests come what may.

If the answer is "it depends", what does it depend on?

  • $\begingroup$ If you perform more than one (t-) test on the same sample then you have a multiple testing problem. This inflates the type I error. You could take a look at this answer: stats.stackexchange.com/questions/164181/… $\endgroup$ – user83346 Aug 31 '15 at 5:05
  • $\begingroup$ @fcoppens is that really true when you have a specific prediction Group 1 < Group 2 < Group 3, like OP does? $\endgroup$ – jona Aug 31 '15 at 19:36
  • $\begingroup$ @jona: if you break that up into more than one test then you have a multiple testing problem. If you perform a ' joint' test for all inequalities at the same time with one test then not. $\endgroup$ – user83346 Sep 1 '15 at 4:31

When you have an apriori alternative hypothesis, you can look directly at doing a hypothesis test specifically for that, that is, an hypothesis test with increased power for the alternative you are interested in. You have an ordered alternative, so could use a test specifically constructed for that. Ordered alternatives also named monotone alternatives. There is a section about that in the old book by Miller: "Beyond ANOVA".

The likelihood ratio test for monotone alternatives was developed by Bartholomew (1959, 1961, refs in Miller), and is connected to isotone regression and the "pool adjacent violators" algorithm. The problem is that the null distribution is complicated, and needs special tables, Miller's book has references. And, there is a complete book: http://www.amazon.com/Selecting-Ordering-Populations-Statistical-Methodology/dp/B010WF5YXK/ref=sr_1_4?s=books&ie=UTF8&qid=1441007651&sr=1-4&keywords=ordering+selecting

There is an alternative, maybe easier to implement (following Miller). This is due to Abelson and Tukey (1963), and summarized by: If the monotone alternative is $\mu_1 \le \mu_2 \le \dots \le \mu_I$ then base the test on the contrast with coefficients $c_1, c_2, \dots, c_I$. Write $R$ for the region in parameter space satisfying $\mu_1 \le \mu_2 \le \dots \le \mu_I$ and choose a maximin approach by choosing the contrast by maximizing the minimum power over $R$. Detail in Miller, above. The paper by Abelson and Tukey: "Efficient Utilization of Non-Numerical Information in Quantitative Analysis: General Theory and the Case of Simple Order", the Annals of Mathematical Statistics, 1963, have a table of maximin contrasts, which we report a part of below: $$ \begin{array}{crrrrrrr} j & n=1 & n=2& n=3 & n=4 & n=5 & n=6& n=7&n=8 \\ \hline 1 &-.707&-.816&-.866&-.894&-.913&-.926&-.935 \\ 2 & .707&.000&-.134&-.201&-.242&-.269&-.289 \\ 3 & & .816& .134&.000&-.070&-.114&-.144& \\ 4 & & & .866&.201&.070& .000&-.045 \\ 5 & & & &.894&.242&.114&.045 \\ 6 & & & & &.913&.269&.144 \\ 7 & & & & & &.926&.289 \\ 8 & & & & & & &.935 \\ \hline \end{array} $$

By using such an approach, there is only one hypothesis test and obviously no multiplicity problems!

A formula for the contrasts in the simple order case $\mu_1 \le \dots \le \mu_n$ is given by: $$ c_j = \{(j-1)[1-((j-1)/n)]\}^{1/2} - \{j(1-j/n)\}^{1/2} $$ and then one can simply use a $t$-test with this contrast. This do not need any special code (or package) in R.

There do seem to exist some implementations in R for tests for ordered alternatives, not the ones I wrote about, but a nonparametric alternative, the Jonckheere-Terpstra test. See https://en.wikipedia.org/wiki/Jonckheere's_trend_test . I will show an implementation in CRAN package clinfun, code below:

library(clinfun) ### From CRAN
set.seed(7*11*13)    ### for reproducibility
mu  <-  c(10,  11,  12)
g  <-   length(mu)
n  <-   10   ### number of obs in each group
gg  <-  list()
for (j in seq(along=mu)) gg[[j]] <- rnorm(n, mu[j],  sd=1.0)
nn  <-  sapply(gg,  length)
ind  <- rep(1:g,  nn)
xx  <-  unlist(gg)
test.ordered  <-  jonckheere.test(xx,  ind,  alternative="increasing")

Output when run is:


    Jonckheere-Terpstra test

JT = 219, p-value = 0.00407
alternative hypothesis: increasing

Other implementations in coin, npsm and the package lawstat has analogous tests for equality of variances against ordered alternatives. I will come back extending this when I get hold of the referenced papers!

  • $\begingroup$ Could you demonstrate how to implement the procedure described in your second to last paragraph in for example R? $\endgroup$ – jona Aug 31 '15 at 19:38
  • 1
    $\begingroup$ Will try, but will need to get hold of the papers first ... $\endgroup$ – kjetil b halvorsen Aug 31 '15 at 19:44
  • 1
    $\begingroup$ I outlined two approaches: 1) t-tests come what may, and 2) t-tests only if ANOVA is significant. Your suggested approach sounds superior to either. However, I still wonder about the merits of the two approaches I outlined, because they're both frequently used within the field in which I work. $\endgroup$ – user1205901 - Reinstate Monica Sep 1 '15 at 3:29
  • $\begingroup$ en.wikipedia.org/wiki/Jonckheere%27s_trend_test is another method, with R implementation. $\endgroup$ – kjetil b halvorsen Sep 2 '15 at 14:36
  • $\begingroup$ See restriktor.org $\endgroup$ – kjetil b halvorsen Mar 27 at 3:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.