Say for a 2x2 experiment, the power analysis said the sample size should be over 1000 to detect 0.1 effect size. Right now, with the sample size of below 100, the effect size of .12 is insignificant with the p value being 0.8 (very far from .05)

Is it fair to say that even if I attain the required over 1000 sample, the result is less likely to be significant?

  • $\begingroup$ I don't think so, no. $\endgroup$ Commented Jul 1, 2023 at 5:02
  • 2
    $\begingroup$ Less likely than what? $\endgroup$
    – Peter Flom
    Commented Jul 1, 2023 at 12:42
  • $\begingroup$ @PeterFlom less likely than those with the P value close to .05 $\endgroup$ Commented Jul 1, 2023 at 13:30
  • $\begingroup$ It's a bit unclear why you want to make such a statement. Is it to "call it a day" and stop collecting data? Or is there another reason? Knowing your reasons may help answering you. $\endgroup$
    – J-J-J
    Commented Jul 2, 2023 at 12:53
  • $\begingroup$ @J-J-J We did an online experiment with less than 100 student sample as students weren't enthusaistic in helping out. The sample size is far from the power analysis required for a medium effect size (which is more than 200). But anyway, I did the analysis as it is and found the manipulation is not significant. And I am being told the manipulation failed regardless of sample size as the P value is like .8 ... I am not sure about it. If that's not true, maybe I will use my own money with crowdsourcing platform.... But I got to be cautious as the money is really from my own pocket.... $\endgroup$ Commented Jul 2, 2023 at 16:48


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.