1
$\begingroup$

BACKGROUND:

I run a 2X2 Between-Subjects ANOVA. I obtained:

  • a low power (< 0.30) for both Independent Variables
  • a not statistically significant result for one of the Independent Variables.

Still, the effect size of the two independent variables were: eta-squared 1,17% for the non statistically significant one and eta-squared 5,64% for the other one.

According to Cohen, these could be considered respectively a small and medium effect sizes. Also, as he said

"a medium effect is visible to the naked eye of a careful observer. A small effect is noticeably smaller than medium but not so small as to be trivial."

QUESTION:

Is there anything valuable that I could take out from this experiment? Or given these results (low power, no statistical significance and medium-low effect size) there's nothing valuable that can be taken?

I'm especially interested with regards to the non statistically significant independent variable: it is "gender" and I was evaluating whether or not it had an impact on measuring investment alternatives. Could I conclude my analysis saying "the experiment shows no difference exists" or better say "the experiment shows that only a subtle difference exists, and a bigger sample size could detect a significant difference"?

Thank you.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

First, you should have done the power analysis before starting the whole process. That would have told you that you either needed more subjects than you actually go or to do a different project, at least if you were hoping to find statistically significant results.

Next, the whole question of whether you can report on statistically nonsignificant results has been debated a lot. Different statisticians have different views. My own view is strongly towards the "yes you can report them" point of view. Others will say no.

But this is probably not the place to get into a long discussion about it.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you Peter. I did run a power analysis, but unfortunately the respondents were lower than expected (112, distributed in 4 equal groups). Would you have any academic paper to recommend with regards to your point of reporting statistically nonsignificant results? $\endgroup$
    – Maurizio Lonas
    Jun 10, 2016 at 12:07
  • $\begingroup$ Searching on "p value controversy" will find a lot of material. One book is "The cult of statistical significance". But be warned that this is not universally accepted (to put it mildly) $\endgroup$
    – Peter Flom
    Jun 10, 2016 at 12:11
  • $\begingroup$ Ok, great! Finally, could be something like this a valuable answer in a paper? What do you think? "The experiment showed a non significant result for gender, one of the two independent variables. However, this could be explained by the low power of the test (<0,30). Still, the small effect size (1%) suggests that a subtle and not trivial relationship between gender and investor evaluation exists". $\endgroup$
    – Maurizio Lonas
    Jun 10, 2016 at 12:21
  • $\begingroup$ Power calculations have to include effect size. So, I think you have some confusions. Whether an effect size of 1% is noteworthy depends on the context of the effect and on the field in general, rather than on statistical considerations. $\endgroup$
    – Peter Flom
    Jun 10, 2016 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.