# Statistical significance, repeatability, and sample size (50 shades of grey)

Reading about the problems with significance I wondered how one would approach an experiment in the right way, considering that in practice the sample size cannot be arbitrarily large.

While some meta-studies deal with confirmation bias and similar effects, I want to focus here on significance. I understand that $p=0.05$ was arbitrarily chosen to lead wrong way in 1 out of 20 trials. Reading about the problem with repeatability (Nature, Science), it seems that the small sample size is the main problem.

## Consider the following example I found:

Matt Motyl (University of Virginia) had the hypothesis that radical leftwing and rightwing supporters would recognize less grey-shading colors. With a sample size of $n\approx 2000$ he had a significance of $p=0.01$.

This suggests that it would be highly significant. Because the team read about the problem with repeatability, they repeated the experiment with $n=1300$ (yielding $.995$ power to detect an effect of the original effect size at $\alpha = .05$) and got $p=0.59$ which obviously changes everything.

## My questions:

1. Now what if you repeat the experiment a third time with $p=0.049$ or lower? Do you have to do it a fourth time? I thought to prevent that, the main idea is the significance value.
2. For this example would an increase of sample size eliminate the problem or what is the right approach to conduct the experiment 1 time only? Or in other words, each clinical study is also done only 1 time and it seems to be OK, what am I missing?
3. In case a meta-study is necessary, why are then meta-meta studies (and so on) not necessary? The meta-study also has a $p$-value.
4. Optional: If you read study results it seems one can deviate if the study used p-hacking or selective publishing. How could one detect selective publishing, as it means THIS experiment was "correct" (just like with the grey-shading study)?
• +1. I edited your question quite a bit, please check that you are happy with my edits. I inserted the link to the paper where this "50 shades of grey" failed experiment is described. I read it before and I find this description disturbingly incomplete and puzzling: the authors don't comment at all on whether they think their original effect size was over-estimated (and p-value under-estimated) because of some amount of unconscious p-hacking, or whether it was pure luck (or lack of it). They don't describe their analysis and do not present their data. So I am not sure what to make of it. Mar 10, 2016 at 11:52
• @amoeba thank you, much clearer than my version =) Mar 10, 2016 at 12:01