9
$\begingroup$

Challenge

In our office, I have been involved in a discussion about sample size and its influence on effect size - can you help me out and explain further?

Base

When conducting a power analysis, one can determine sample size for a specific effect size in a specific design.

Problem/Discussion

What happens if the a priori determined sample size is exceeded (e.g. determined sample in power analysis was $N=100$, but we could obtain $N=1000$)?

Position 1: Large sample sizes chop up/ destroy effect sizes. When using larger samples than determined in power analysis, danger occurs that "everything becomes significant" (even minor, practically irrelevant effects). Therefore, we should rely to determined sample from power analysis. Doing so, we can reveal "real/relevant" effects.

or

Position 2: Determination of sample size is referring to the minimal sample size which is required to reveal a given effect. Large sample sizes are beneficial, e.g. because of decreasing measurement error. Therefore, real effects can be revealed more easily. Post hoc effect-size calculations offer information about relevance of the effect.

or

Position 3: Position 1 versus position 2 are depending on the study design (e.g. position 1 for t-Test because of seeking for "relevant effects", but position 2 for CFA/ SEM to get more stable, reliable results).

or

Position 4: Another possible position for an alternative explanation.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
13
$\begingroup$

the danger occurs that "everything becomes significant" (even minor, practically irrelevant effects).

This is not an argument against large sample sizes, it's a direct argument against hypothesis testing for your particular problem.

If you have a problem rejecting for small effect sizes don't use ordinary hypothesis tests.

It may be that you need an equivalence test (or perhaps a noninferiority test).

It may be that you need an interval estimate of the effect size (i.e. a confidence interval).

It may be that you need something else.

This also relates to Position 3. If you have a notion of a "relevant effect" you should not be using ordinary hypothesis tests.

If your position is not that more power is better, stop using those hypothesis tests. It's not the correct tool for that job.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ Equivalence tests... I have to get more info on this. It makes a lot of sense... $\endgroup$
    – Antoni Parellada
    Commented Jan 6, 2016 at 13:39
  • 1
    $\begingroup$ What are some good sources to learn about equivalence tests? I haven't found much online. $\endgroup$
    – Clarinetist
    Commented Jan 6, 2016 at 14:39
  • 1
    $\begingroup$ My company was approached by another company requesting feedback on a proposed study design. They attempted to use a two-sided test of the form $$H_0 : \mu_1 - \mu_2 = 0 \quad \text{vs.} \quad H_a : \mu_1 - \mu_2 \ne 0$$ to show that that there was no difference in means. When I pointed out to them on two separate occasions why this was blatantly wrong, and showed them that the regulatory agency overseeing their conduct was aware of this flaw, they ignored me. $\endgroup$
    – heropup
    Commented Jan 6, 2016 at 16:30
  • 1
    $\begingroup$ To make claims of equivalence, then, the structure of the hypothesis test could be (for testing differences in means) $$H_0 : |\mu_1 - \mu_2| \ge \Delta \quad \text{vs.} \quad H_a : |\mu_1 - \mu_2| < \Delta,$$ where $\Delta$ is some equivalence margin that is regarded as a standard of acceptable similarity. Note we cannot simply flip around $H_0$ and $H_a$. I leave it as an exercise to the reader to figure out why. $\endgroup$
    – heropup
    Commented Jan 6, 2016 at 16:35
  • 2
    $\begingroup$ @Glen_b could you mention some references to learn more about equivalence tests? Yet, I only found some papers. $\endgroup$
    – Jens
    Commented Jan 8, 2016 at 14:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.