2
$\begingroup$

I'm at the point of conducting a 2x2 factorial design. My adviser says that it has no use to conduct an experiment based on such a simple design unless you expect everything to be significant; 2 main effects and 1 interaction effects. I think that even when you expect no significant effects, they are still interesting to know. Isn't there also a need to know if something really does not have an effect instead of testing only on whether something has an effect? Moreover, he proposed that interaction effects are most important in a factorial design. So these you need to expect no matter what to be significant.

He suggested to include another factor to make it a 2x2x2 design in order to have more interaction effects. To me his comments sound kind of odd. I have reasons why I expect something to be not significant. If I go with his argumentation, non significant effects should never be tested? Like I said before, I think expecting not significant effects are just as interesting as testing for significant effects.

Since he's the professor and I'm not, any second opinion on this?

$\endgroup$

1 Answer 1

1
$\begingroup$

What's interesting or not is something that should be decided based on what you know about the topic. It's perfectly fine to run a 2x2 experiment, use only a single manipulation or even study the distribution of a single variable (with no experimental manipulation or bivariate statistics at all) if that variable present some theoretical or practical interest.

One issue here is that you seem to be overly concerned about significance testing. In fact, it's often not reasonable to expect any effect to be exactly zero (this is obviously true and has been stressed many times in relation to observational research – modeling in economics, sociology, political science, etc. – but it is also true for psychology experiments although perhaps not always in chemistry of physics).

It's also incorrect to consider that failing to reject the null hypothesis is strong evidence for the absence of any effect as the power to detect a given effect depends on the sample size and the actual magnitude of this effect. So if you actually expect some variable not to have an effect and you are specifically interested in that, just running an ANOVA in the hope that the p-values are above the usual threshold is not the right strategy, no matter how many factors you have in your design.

For all these reasons, your adviser has a point in the sense that blindly running a 2x2 experiment and finding that no effect is significant might very well leave you with uninterpretable and unpublishable data. Adding some manipulation that appear to “cancel” an effect (as shown by a simple effect and a significant interaction) could be a way to “sneak in” the story about the absence of the effect under some condition but the fact remains that a non-significant effect alone is difficult to interpret and next to impossible to publish in some journals/disciplines.

However, I think the solution here is not necessarily to add a few other factors but to think hard about effect size and formulate more sophisticated hypothesis than “something or other has an effect”. You can then use power analysis, equivalence testing, or precision in parameter estimates to ensure that your experiment provides valuable information on this hypothesis in any case.

Also, in some fields like psychology, researchers work very hard to bring everything to a 2x2x… ANOVA design even when some variables naturally have a quantitative or continuous interpretation. Another way to make an experiment more informative is to use several levels for such independent variables or try to directly model the relationship between these quantitative variables and your response.

Finally, one consideration would be the cost of additional conditions. In survey research, common wisdom is that people don't mind long questionnaires, the difficulty is getting them to participate at all. In that case, researchers often add variables/questions “just in case”, to address ancillary hypotheses or get more out of the effort. At the other extreme, if the study is necessarily between-subject and adding participants is very costly (say brain imaging with hard-to-recruit patients), you would need to think very carefully before adding any factor or condition.

$\endgroup$
4
  • $\begingroup$ I could agree if I would have given an answer from my adviser about effect sizes, like you did. It's just some general thoughts about what I'd roughly expect. What striked me was that he just said that I shouldn't do a 2x2 if I didn't expect "good", and with that he means strong significant, results and that I should add more factors. Especially for the interaction effects. To me every result is useful whether it's significant or not. It says something about what you have found. Like you said; there's much more than significant levels. Unfortunately statistics courses usually don't cover that $\endgroup$ Commented Jul 25, 2013 at 12:59
  • $\begingroup$ Well maybe I wasn't very clear, a non-significant result does not tell you much, no. Knowing that an effect is small is interesting but testing for a difference and failing to reject the null hypothesis is not the right way to establish that. $\endgroup$
    – Gala
    Commented Jul 25, 2013 at 13:31
  • $\begingroup$ I think I understand what you mean, maybe I wasn't very clear myself. I agree that a not significant result by itself doesn't say much, but the same holds for significant results. I talk about significance here assuming that the results contain useful information and can draw steady conclusions. What I mean is that my adviser just doesn't want non significant result even when it contains valuable information. He just want to have significant results no matter what. Thereby is rejecting my design and saying 2x2 is not the way to go. It just makes me doubt about myself because I think otherwise $\endgroup$ Commented Jul 25, 2013 at 14:13
  • $\begingroup$ You don't seem to actually agree or even read what I am writing. You should definitely not aim for “non-significant” tests, they do not “contain valuable information”. Your advisor is right in that respect. I would add that significant differences do not contain that much information either but at least they have a clear interpretation and stand some chance of being published. If you are specifically interested in showing that two conditions are similar either plan a test for equivalence or eschew testing entirely but again do not plan a study aiming for a non-significant result. $\endgroup$
    – Gala
    Commented Jul 25, 2013 at 14:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.