3
$\begingroup$

I counducted a mixed model ANOVA with time as the within subjects variable (3 levels) and group as the between subjects variable (2 levels). All is significant (time, group, and timeXgroup). To test if differences over the three times are significant in both groups, I conducted two separate repeated measures ANOVAs, one for each group, with time (3 levels) as the withing subjects variable.

I have a problem with the control group, because ANOVA for this group is slightly significant (p = 0.044) but the multiple comparisons with Bonferroni correction are all not significant... Indeed, in the graph I can see that the differences across time are not very big. However I don't know how to explain this result... Can you help me?


Edit:
I try to explain myself better so maybe you can understand my situation. These are the descriptive statistics:

Control group (N=10)

Mean (SD)
1st measure 2.40 (1.36)
2nd measure 2.87 (1.57)
3rd measure 2.55 (1.48)


Experimental group (N=10)

Mean (SD)
1st measure 3.04 (1.14)
2nd measure 6.71 (3.52)
3rd measure 4.64 (1.76)

As you can see the control group has a small change over time, the experimental group instead has a big change. I ran the mixed 2x3 ANOVA to see if the two groups were significantly different and if there was an interaction. Since all was significant, I wanted to see if both group differ significantly over time or only the experimental group. Thank you!

$\endgroup$
  • $\begingroup$ This problem really happens when testing hypothesis (lack of logical coherence among the results). This particular problem you had is known as lack of coherence (or monotonicity). See for example this $\endgroup$ – madness Nov 18 '12 at 15:58
  • $\begingroup$ Thanks for your answer, I had a look at the article you suggest but I'm not a statistician and I don't understand very well. I would need some help to interpret these results in light of my research, e.g. what should I say about the control group? Did this group change over time or is an error? $\endgroup$ – Federico Nov 18 '12 at 16:20
  • $\begingroup$ I guess that's extremely hard to interpret. How are the other p-values? Maybe you should just say something like "if there is a difference it is probably here and here". Or you couldn't perform a different test (instead of a mixed model ANOVA) and compare the results. I don't have a suggestion of a test, but I'm pretty sure other people will have good ideas. $\endgroup$ – madness Nov 18 '12 at 16:41
  • $\begingroup$ Thank you. Actually for my research purpose it's not very important to have a significant change over time in control group. With this result, would be better to say "There is a change, but it is barely significant (p = 0.044), in fact the multiple comparisons are all not significant" or "The ANOVA result is probably wrong"? $\endgroup$ – Federico Nov 18 '12 at 18:03
  • $\begingroup$ The ANOVA result is not wrong in the statistical sense, this can really happen. That's a problem that happens some times, results from statistical tests can be incoherent. I'd conclude the there is not difference, but if possible I'd just perform another test to have more evidence. $\endgroup$ – madness Nov 18 '12 at 18:25
3
$\begingroup$

Even without Bonferroni corrections ANOVA's do not guarantee any two means are different. For example, in a statistically decisive ANOVA result could come from two pairs of means are different from each other while no individual mean comparison is significant.

Consider why you run an ANOVA. You do it because if you did all of the comparisons with a categorical predictor value then you'd run into a multiple comparisons problem. But then you go and do many of the comparisons... why? The ANOVA means that the pattern of data you see is meaningful. Describe the pattern of data, both in a figure and text, and convey what your data mean. If you really wanted to run all of the multiple comparisons then running the ANOVA was pointless. Also, keep in mind that "all of the comparisons" does not mean just those comparisons between individual means but all of the patterns patterns and combinations you could test, the ANOVA is sensitive to them too.

In your particular case, what you would do is write something like the following. There was a main effect of group, with higher scores in the experimental group and a main effect of time with the first time the lowest score, followed by the last time and finally the highest score was at the intermediate time. However, each of these main effects was qualified by an interaction. The effect of time depends on which group you are in, being greater in the experimental than the control group.

That's what your ANOVA and summary statistics say. Unless there's something more than that you want to say there's no point in running comparisons.

ASIDE: While the following is important, I consider it an aside because the primary question here is interpreting your ANOVA. Your experimental group time 2 variance is so much higher than the others that you're violating assumptions of the ANOVA. You could run simulations to see how much that affects alpha or power in your case. I did a quick one and it shows alpha is generally about 0.06 (if you select 0.05) for each test, sample code below:

nsamp <- 2000
n <- 10
sds <- rep(c(1.36, 1.57, 1.48, 1.14, 3.52, 1.78), n)
x1 <- factor(rep(1:2, times = n, each = 3))
x2 <- factor(rep(1:3, 2*n))

Y <- replicate(nsamp, {
    y <- rnorm(6 * n, 0, sds)
    #y <- rnorm(6*n) # comment out the line above and comment in this one to see what would happen if variances were equal
    m <- aov(y ~ x1 * x2)
    sm <- summary(m)
    ps <- sm[[1]]$'Pr(>F)'
    ps
    #min(ps, na.rm = TRUE)
})

sum(Y[1,] < 0.05)/nsamp
sum(Y[2,] < 0.05)/nsamp
sum(Y[3,] < 0.05)/nsamp
$\endgroup$
  • $\begingroup$ Thank you John for your answer, I edited my question above adding more informations as you suggested me. $\endgroup$ – Federico Nov 18 '12 at 21:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.