0
$\begingroup$

I hypothesised I would find a main effect of X which I did (p=0.028), however, upon using the bonferroni correction my alpha dropped to 0.0125, making this result no longer significant.

I found a partial eta-sqaured effect size of 0.194 which suggests a large effect size. I also used JASP to investigate Bayes factors and found BF10=2.165, which suggests anecdotal support for the hypothesis.

However I am now unsure how to interpret these three values in conjunction, as although the p-value is now not significant, it has a large effect size and the Bayes factor provides some support.

Any help would be greatly appreciated!

$\endgroup$
0
$\begingroup$

When following a strict frequentist approach, there is just one interpretation: "When controlling the familywise type I error rate main effect was not statistically significant." I.e. we failed to "demonstrate" that there is an effect as per the "statistically significant with this familywise type I error rate" standard.

This does not rule out the possibility that there is an effect. In fact, the observed point estimate is (presumably) in the hypothesized direction, just smaller than assumed when planning the experiment - which could be bad luck due to sampling variation, or could be due to there being no effect/a near-zero effect or even an effect in the opposite direction.

$\endgroup$
  • $\begingroup$ Thank you for your response! In my discussion would it be incorrect to write that although the result was insignificant, due to the bayes factor and the fact the p-value was still below a=0.05, this suggests there could a small effect but more research is needed, or something along those lines? $\endgroup$ – Holly Miller Feb 9 at 10:54
  • $\begingroup$ It is correct that this does not exclude a small effect (concluding "no effect proven" is in fact be a misinterpretation). It depends on the power of the study for "small", "medium", or "large" true effect sizes. Perhaps results could have been statistically significant with a "small" effect size, perhaps you got a non-siginficant result with a CI that includes (or point estimate that is a) large effect size. Many may object to overemphasizing a BF or unadjusted p to argue this, when it's really just saying that there is nothing magically changes when p just below 0.05 vs. just above 0.05. $\endgroup$ – Björn Feb 9 at 11:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.