0
$\begingroup$

I keep reading "effect size is independent of sample size" on the interwebs. Don't get me wrong, I get the practical vs statistical significance considerations. What I don't get is that;

-power,significance-level, sample size and effect size are all linked functionally. Fix three and you can determine the fourth.

The point is that if you fix the power and the significance-level and you increase the sample size; the effect size will also functionally change. This trivial exercise can easily be verified in GPOWER.

So researchers with little knowledge and plugging values into the likes of GPOWER can get suspicious outcomes. If a ridiculously large sample size was entered in combination with reasonable power and significance-levels (0.8 and 0.05), then the effect size would mathematically be tiny. As an effect size represents "practical significance", this would show how such a situation can be manipulated.

So why is "effect size" always referred to as being "independent of sample size" when functionally it clearly is actually dependent?

I feel the expression I keep reading should rather be stated:

effect size should be designed to be independent of sample size (or similar)

Improve this question
$\endgroup$
1
  • $\begingroup$ If you fix others, what you get is the minimum effect size that your study can detect as statistically significant at a given level of power. This is not the same as the effect size you will observe from your study which is "independent" of sample size. $\endgroup$
    – Heteroskedastic Jim
    Commented Nov 1, 2018 at 10:36

1 Answer 1

Reset to default
2
$\begingroup$

Welcome to Cross Validated, nevermind. (Ha! I like the ambiguity of that.)

The _observed_effect size is whatever is observed and it is independent of the sample size. The sample size that can pop out of the power analysis is a predicted or theoretical effect size that would allow the equations to balance. In practice it is the observed effect size that matters.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I think one of the main issues regarding effect size (ES) is that researchers refer to Cohen's small, medium and large (0.2,0.5,0.8 for example) and say "I want to be able to detect a small effect size, so I will use ES=0.2" and plug into a program like Gpower...with no reference to prior studies, knowledge of the underlying data. In this way there was no calculation of a practical ES. If the realistic ES was more like 0.8 then the statistical test is invariably significant because it was simply oversampled and the precision of the test exceeds a practical result. $\endgroup$
    – nevermind
    Commented Nov 1, 2018 at 4:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.