0
$\begingroup$

I have a question. If I found the effects of a variable is insignificant (p>.05), but the effect size d is 0.2 (small effect).

So I used GPower to calculate the power and I found it is 0.14, which means I need a larger sample size to get a significant results.

So is the hypothesis rejected or supported? because it will be significant if i have a larger sample size.

Thanks

Let me make it clear. My hypothesis is that priming improves creative performance. So if p>.05, d=0, the hypothesis is rejected, priming has no effect on creative performance. My question is if p>.05, d=0.2 (small effect), is the hypothesis rejected or supported?

$\endgroup$
2
  • $\begingroup$ Any hypothesis test will be significant if you have a larger sample... $\endgroup$ Jun 28, 2014 at 1:50
  • 1
    $\begingroup$ "because it will be significant if i have a larger sample size" - with a continuous outcome, there will always be some estimated non-zero effect size ... so you could always come to the same conclusion as you just did. Does an argument that says "my hypothesis is always supported" make sense? If so, why bother with any sample at all? $\endgroup$
    – Glen_b
    Jun 28, 2014 at 10:10

1 Answer 1

3
$\begingroup$

You did not reject your hypothesis.

But here you have the opportunity for a few insights:

  1. Any non-zero effect size is significant with a large enough sample.

  2. One might interpret the significance of results, absent other considerations, as pretty meaningless beyond "will my editor accept these results?"

  3. One might take the approach that, a priori an effect size needs be be so large in order for one to consider it relevant.

    3a. One can combine tests for difference with tests for equivalence to produce inferences within a hypothesis testing framework that differentiate relevant difference from trivial difference (i.e. an over-powered test) from equivalence from indeterminacy (i.e. an underpowered test).

$\endgroup$
10
  • $\begingroup$ Thank you for your answer. I have two variables. Both of them are insignificant, but one of the effect size is 0.2, another one is 0. So the hypothesis with effect size 0 is rejected. The hypothesis with effect size 0.2 is supported. Is it what you meant? $\endgroup$
    – Perry
    Jun 28, 2014 at 1:11
  • $\begingroup$ No! :) First, you did not reject either hypothesis. Second, how would you answer this question: what is the smallest effect size that matters to you? $\endgroup$
    – Alexis
    Jun 28, 2014 at 1:20
  • $\begingroup$ The smallest effect size that matters to me is 0.2. It means that variable has a small effect on the dependent variable, even if it is statistically insignificant. $\endgroup$
    – Perry
    Jun 28, 2014 at 1:26
  • $\begingroup$ I don't understand why the hypothesis is not rejected if p>0.05 and the effect size is 0. $\endgroup$
    – Perry
    Jun 28, 2014 at 1:30
  • 1
    $\begingroup$ Nope. This is my last comment, since we do not seem to be communicating. $\endgroup$
    – Alexis
    Jun 28, 2014 at 14:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.