Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Challenge

In our office, I have been involved in a discussion about sample size and its influence on effect size - can you help me out and explain further?

Base

When conducting a power analysis, one can determine sample size for a specific effect size in a specific design.

Problem/Discussion

What happens if the a priori determined sample size is exceeded (e.g. determined sample in power analysis was $N=100$, but we could obtain $N=1000$)?

Position 1: Large sample sizes chop up/ destroy effect sizes. When using larger samples than determined in power analysis, danger occurs that "everything becomes significant" (even minor, practically irrelevant effects). Therefore, we should rely to determined sample from power analysis. Doing so, we can reveal "real/relevant" effects.

or

Position 2: Determination of sample size is referring to the minimal sample size which is required to reveal a given effect. Large sample sizes are beneficial, e.g. because of decreasing measurement error. Therefore, real effects can be revealed more easily. Post hoc effect-size calculations offer information about relevance of the effect.

or

Position 3: Position 1 versus position 2 are depending on the study design (e.g. position 1 for t-Test because of seeking for "relevant effects", but position 2 for CFA/ SEM to get more stable, reliable results).

or

Position 4: Another possible position for an alternative explanation.

share|improve this question
up vote 11 down vote accepted

the danger occurs that "everything becomes significant" (even minor, practically irrelevant effects).

This is not an argument against large sample sizes, it's a direct argument against hypothesis testing for your particular problem.

If you have a problem rejecting for small effect sizes don't use ordinary hypothesis tests.

It may be that you need an equivalence test (or perhaps a noninferiority test).

It may be that you need an interval estimate of the effect size (i.e. a confidence interval).

It may be that you need something else.

This also relates to Position 3. If you have a notion of a "relevant effect" you should not be using ordinary hypothesis tests.

If your position is not that more power is better, stop using those hypothesis tests. It's not the correct tool for that job.

share|improve this answer
1  
Equivalence tests... I have to get more info on this. It makes a lot of sense... – Antoni Parellada Jan 6 at 13:39
1  
What are some good sources to learn about equivalence tests? I haven't found much online. – Clarinetist Jan 6 at 14:39
1  
My company was approached by another company requesting feedback on a proposed study design. They attempted to use a two-sided test of the form $$H_0 : \mu_1 - \mu_2 = 0 \quad \text{vs.} \quad H_a : \mu_1 - \mu_2 \ne 0$$ to show that that there was no difference in means. When I pointed out to them on two separate occasions why this was blatantly wrong, and showed them that the regulatory agency overseeing their conduct was aware of this flaw, they ignored me. – heropup Jan 6 at 16:30
1  
To make claims of equivalence, then, the structure of the hypothesis test could be (for testing differences in means) $$H_0 : |\mu_1 - \mu_2| \ge \Delta \quad \text{vs.} \quad H_a : |\mu_1 - \mu_2| < \Delta,$$ where $\Delta$ is some equivalence margin that is regarded as a standard of acceptable similarity. Note we cannot simply flip around $H_0$ and $H_a$. I leave it as an exercise to the reader to figure out why. – heropup Jan 6 at 16:35
1  
@heropup They're typically structured as a pair one-sided tests, but I presume you know that already. – Glen_b Jan 6 at 16:43

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.