Reading about the problems with significance I wondered how one would approach an experiment in the right way, considering that in practice the sample size cannot be arbitrarily large.
While some meta-studies deal with confirmation bias and similar effects, I want to focus here on significance. I understand that $p=0.05$ was arbitrarily chosen to lead wrong way in 1 out of 20 trials. Reading about the problem with repeatability (Nature, Science), it seems that the small sample size is the main problem.
Consider the following example I found:
Matt Motyl (University of Virginia) had the hypothesis that radical leftwing and rightwing supporters would recognize less grey-shading colors. With a sample size of $n\approx 2000$ he had a significance of $p=0.01$.
This suggests that it would be highly significant. Because the team read about the problem with repeatability, they repeated the experiment with $n=1300$ (yielding $.995$ power to detect an effect of the original effect size at $\alpha = .05$) and got $p=0.59$ which obviously changes everything.
- Now what if you repeat the experiment a third time with $p=0.049$ or lower? Do you have to do it a fourth time? I thought to prevent that, the main idea is the significance value.
- For this example would an increase of sample size eliminate the problem or what is the right approach to conduct the experiment 1 time only? Or in other words, each clinical study is also done only 1 time and it seems to be OK, what am I missing?
- In case a meta-study is necessary, why are then meta-meta studies (and so on) not necessary? The meta-study also has a $p$-value.
- Optional: If you read study results it seems one can deviate if the study used p-hacking or selective publishing. How could one detect selective publishing, as it means THIS experiment was "correct" (just like with the grey-shading study)?