3
$\begingroup$

I have used a 2x3 mixed between-within subjects ANOVA to assess the impact of three feedback conditions (Grade, Comment and No Feedback) on participants’ scores on the Specific-Self Efficacy Scale on two occasions, pre- and post- task.

I did not get a significant main effect but I did get a significant interaction effect.

My tutor has told me that I can interpret the results from the interaction graph alone and that there is no need to run tests to see if there are significant differences between pre and post task scores in each of the feedback conditions.

However, when discussing these results in my manuscript, I do not feel confident in relying on a graph alone.

Is it appropriate to only use the interaction graph to discuss results or we you recommend running some paired samples t-tests?

Thank you for taking the time to read my query!

$\endgroup$
1
$\begingroup$

I like your tutor, but many people will not. The question of effect size (which shows up on a graph) vs. statistical significance (which you get as part of the output from a test) has been discussed a lot; I am strongly on the effect size side, but others differ (and not completely unreasonably, either).

However, it is often nice to be able to put some precise numbers in the text and it is hard to get those from a graph alone.

$\endgroup$
  • $\begingroup$ Thank you very much, Peter ! I think I will include some precise numbers in the text. $\endgroup$ – NMcAdden Mar 9 '13 at 13:40
0
$\begingroup$

You should also note that since you have already run an omnibus ANOVA, any subsequent analyses are being done post-hoc. You should run an appropriate multiple comparison procedure when looking for where the significant differences lie to control your family-wise Type I error rate.

For example, you could use the multcomp package in R, and then use glht() on the model object.

$\endgroup$
  • $\begingroup$ OK, that is good to know ! Thank you ! I am familiar with Bonferroni adjustments from our statistics clinics, is that appropriate here? I would have to divide my alpha level of 0.05 by three, making the new alpha level .017? $\endgroup$ – NMcAdden Mar 9 '13 at 18:19
0
$\begingroup$

What you do in this situation really depends on whether you're testing hypotheses that you had a priori or whether you are conducting exploratory tests. If you had a strong a priori hypothesis that, for example, people who view comments would increase in self-efficacy from pre-test to post-test relative to people in the other two conditions, then you would be reasonably justified in testing the following two contrasts:

Contrast 1:

  • Grade: -1

  • Comment: 2

  • No feedback: -1

Contrast 2:

  • Grade: -1

  • Comment: 0

  • No feedback: 1

Contrast 1 tests the difference between the comment condition and the average of the no feedback and grade conditions, while contrast 2 tests the grade condition vs the no feedback condition. Contrast 1 would test your focal hypothesis, while contrast 2 would represent a "residual" contrast that tests whether your data contain a pattern that is different from the one that you hypothesized.

If you're interested in whether these comparisons differ from pre-test to post-test, you would test for an interaction between each contrast and time. In a situation where you have strong a priori hypotheses, you might even be justified in testing these contrasts without testing the omnibus interaction, since the omnibus interaction is a very weak test of your effects of interest in this case. See Abelson and Prentice (1997) for a more in-depth discussion of this method of testing hypotheses involving categorical variables with three or more levels.

If, on the other hand, you are conducting mostly exploratory post-hoc tests, then you probably still want to conduct some follow-up tests, but you should almost certainly apply some post-hoc corrections to account for the fact that these tests are exploratory. There are a lot of post-hoc corrections available, but one of the most flexible is Holm-Bonferroni. Holm-Bonferroni is just as flexible as Bonferroni, but is uniformly more powerful.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.