1
$\begingroup$

I have run a psychology experiment where participants received one of three sets of instructions (between-subjects). Either they were told to approach the experiment a certain way, told to approach it a different way or it was not specified how they should approach the task (it turned out that some in this group did it one way and some the other, about 60/40).

My ANOVA on performance in the experiment is nonsignificant, but my comparison between the 2 groups which received specified instructions were significant either as a planned comparison or in an independent t-test (the effect size between the two is pretty good, Cohen's d = ~0.45). I've been taught not to interpret ANOVA comparisons unless the overall effect is significant, but it seems to me that including this middle group (the means of which tend to be between the two groups) is underpowering the ANOVA. Is it "cheating" to just compare the two groups with a t-test? It certainly doesn't seem right to conclude there is no difference between groups when the ANOVA only breaks with inclusion of this middle group.

Improve this question
$\endgroup$
4
  • $\begingroup$ What's the 95% CI on the d, and what's the p of the t-test? $\endgroup$
    – jona
    Commented Sep 6, 2013 at 10:31
  • 3
    $\begingroup$ Note that a non-significant p-value doesn't justify a conclusion of "no difference between groups". $\endgroup$
    – mark999
    Commented Sep 6, 2013 at 10:43
  • 1
    $\begingroup$ I think you were taught badly (although it's a common way of teaching). People rely on p-values too much and on effect sizes too little. $\endgroup$
    – Peter Flom
    Commented Sep 6, 2013 at 12:08
  • $\begingroup$ Yeah sorry about that Mark, and I agree Peter $\endgroup$
    – luke123
    Commented Sep 8, 2013 at 7:02

1 Answer 1

Reset to default
1
$\begingroup$

The comments already raised many important issues and provided good general advice (don't overinterpret non-significant p-values, don't think that dichotomous significant/non-significant decisions is all there is to data analysis and focus on effect size). Previous questions on post-hoc tests (e.g. What do we call multiple testing?) might also shed some light on the advice you received about multiple comparisons.

However, your experiment seems a little unusual in that the third group is really a mixture of participants who self-assigned to one of the two other groups. Beyond the statistical issues related to multiple comparisons, double dipping and the like, ignoring the “no instruction” condition would seem defensible in this particular case but it begs the question of why you included it in the experimental design in the first place.

If you knew that participants were going to choose one of the two strategies, it was obvious that the variability among participants who did not receive any instruction was going to be higher and their mean score fall somewhere between the two other groups. What a test on these data could possibly reveal is not clear to me.

Since it seems you have a measure of the approach each participant took, another practical way to approach this problem would be to create a variable representing this strategy and compare the two groups defined by this variable. One drawback is that since the strategy is measured and not randomly manipulated, your data are not purely experimental and a causal interpretation of the effect would be much more questionable.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.