0
$\begingroup$

I know that the power of an experiment/study is determined by significance value $\alpha$, the effect size, and the sample size. The power of an experiment increases if either of those increase. And both an underpowered as well as an overpowered experiment are bad.
I understand the problem with underpowered experiments: if $\alpha$ is too low we may not find anything, if the effect size is too small then $H_0$ and $H_1$ are so close they may be the same distribution. And to small a sample size, that's obviously bad they may not represent their population good enough.

I've encountered 3 reasons why an overpowered experiment is bad: it takes up too many samples which is expensive or unethical and "tiny effects may create a significant result e.g. the fact that we measured on a Monday". I only care about the last one.

I understand how a too large $\alpha$ would be bad, by definition we would have a higher probability of falsely finding an effect.
Assuming the effect size to be higher than it is would mean to overestimate the significance of our findings, that makes sense. But a larger actual effect size shouldn't be a problem, should it?
Now for the sample size I have no explanation. If anything, I would expect more measurements to represent the population better! I would expect more measurements to even out small biases!
What am I getting wrong here?

$\endgroup$
1
$\begingroup$

Assuming the effect size to be higher than it is would mean to overestimate the significance of our findings, that makes sense. But a larger actual effect size shouldn't be a problem, should it?

Effect sizes can be seen as a "signal to noise" ratio. For example, in paired t-test the signal part is the mean difference and the noise part is the standard deviation of the difference. Let's say a drug can reasonably lower a certain bad cholesterol by 5 units (SD 1 unit). If we just look at effect size and ignore sample size and alpha for now, an "overpowered" experiment that can detect a large effect size could have set an effect so unrealistically high (say, 50 units difference) that the current sample would have 99.99% power to detect it, but the drug will never be that effective. So, in a way, it's close to a circular reasoning: a study that is overpowered to detect a difference of 50 units is more like an under-powered study, given the true difference is close to 5 units.

Now for the sample size I have now explanation. If anything, I would expect more measurements to represent the population better! I would expect more measurements to even out small biases! What am I getting wrong here?

For this question it seems your question has already provided the answer:

it takes up too many samples which is expensive or unethical and "tiny effects may create a significant result

So, I am not sure what is not clear.

$\endgroup$
  • 2
    $\begingroup$ I am aware of the dogma: too large sample size -> "tiny effects may create a significant result". But I want to understand the reasoning behind it. $\endgroup$ – user2740 Nov 14 '17 at 13:46
1
$\begingroup$

The real problem isn't that "overpowered" experiments may reveal tiny significant effects. The problem is that many academic fields, as well as popular science reporting, emphasize statistical significance over effect size (or practical significance). Given this culture, it can be problematic for experiments to have very large sample sizes, because this allows them to detect very small effects that are of no practical relevance, but which may now be published and discussed with undue prominence.

Of course, the proper solution to this is not to avoid large sample sizes, but rather to report and take notice of effect sizes. It's okay if you find many tiny significant effects in your data; you can still discern those effects that are actually important from those that merely exist and don't have much influence.

$\endgroup$
0
$\begingroup$

The problem with 'overpowering' only occurs if you completely base your conclusions on the black-and-white distinction/cut-off of a p-value threshold, without considering effect size, the relevancy of the effect, the bias in the sample/sampling method, etc.


Think about it this way, if you have a very large sample, small effects might also have small variation. Consequently, as test statistics for most tests (e.g. t-tests) are some variant of effect size divided by measure of variation ($teststat = \frac{effect size}{variation}$); and variation is often determined as the sum of (squared) differences between each measurement i and some central statistic (e.g. the mean), divided by a function of the sample size ($variation = \frac{\sum\limits_{i=1}^n(X_i-\hat{X})^2}{\sqrt{n}}$), you can see how increasing sample size could increase a test statistic into a 'statistically significant' area, by decreasing variation due to increasing $n$ (sample size).

To come round to my opening statement: if then, you completely disregard all other information except for the p-value corresponding to the resulting test-statistic, you can easily see that for any single effect size, there is a sample size at which that effect size becomes statistically significant while keeping the alpha level and sampling population constant. Finally, this would mean that you can 'conclude' basically anything (and thus practically nothing) if you only look at the p-value, without taking into account sample size, effect size and study design (where the latter I now use to denote bias, violated test assumptions, and incorrect test use, etc...)


On a slightly different note, If you reverse the aforementioned 'formula's' this is how 'powering' something statistically is recommended to be done. The goal then is to get an idea of the required sample size. However, this should be done before performing a study/obtaining your sample, as a way to estimate the required sample size based on assumptions and expectations.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.