Statistical significance, repeatability, and sample size (50 shades of grey)

Question

Reading about the problems with significance I wondered how one would approach an experiment in the right way, considering that in practice the sample size cannot be arbitrarily large.

While some meta-studies deal with confirmation bias and similar effects, I want to focus here on significance. I understand that $p=0.05$ was arbitrarily chosen to lead wrong way in 1 out of 20 trials. Reading about the problem with repeatability (Nature, Science), it seems that the small sample size is the main problem.

Consider the following example I found:

Matt Motyl (University of Virginia) had the hypothesis that radical leftwing and rightwing supporters would recognize less grey-shading colors. With a sample size of $n\approx 2000$ he had a significance of $p=0.01$.

This suggests that it would be highly significant. Because the team read about the problem with repeatability, they repeated the experiment with $n=1300$ (yielding $.995$ power to detect an effect of the original effect size at $\alpha = .05$) and got $p=0.59$ which obviously changes everything.

See Nosek, Spies, and Motyl, 2013, Scientific Utopia II. Restructuring Incentives and Practices to Promote Truth Over Publishability (open access).

My questions:

1. Now what if you repeat the experiment a third time with $p=0.049$ or lower? Do you have to do it a fourth time? I thought to prevent that, the main idea is the significance value.
2. For this example would an increase of sample size eliminate the problem or what is the right approach to conduct the experiment 1 time only? Or in other words, each clinical study is also done only 1 time and it seems to be OK, what am I missing?
3. In case a meta-study is necessary, why are then meta-meta studies (and so on) not necessary? The meta-study also has a $p$-value.
4. Optional: If you read study results it seems one can deviate if the study used p-hacking or selective publishing. How could one detect selective publishing, as it means THIS experiment was "correct" (just like with the grey-shading study)?

Answer 1

You pose several important questions, some focusing on hypothesis testing, some on multiplicity, and so forth. These are my answers:

1. The typical approach is to repeat the experiment analyzing every time as no prior study has been conducted. So in a standard frequentist framework only this eventual p counts. I find it wrong and wasteful, as a frequentist meta-analysis or a Bayesian synthesis would borrow information from prior studies as well. Notice indeed that a Bayesian meta-analysis without informative priors and a frequentist meta-analysis encomppassing the same studies will provide very similar, if not identical, inferential estimates. Yet, take notice that the typical Food and Drug Administration approach for regulatory approval of drugs typically disregards prior studies for hypothesis testing.

2. The problem is not simply sample size, it mainly has to do with the precision of the effect and the effect size you think meaningful. A drug which reduces blood pressure by 0.001 mm Hg might be shown efficacious in a mega-trial, but this statistical significance would not be clinically meaningful (that is why ASA and many others are pushing for the abolition of p values and a transition to other approaches, such as confidence intervals). In any case, in a typical frequentist framework, a study could only test a single hypothesis (single test thus), per study, with all the other analysis requiring some penalization for multiplicity.

3. You may indeed perform a meta-analysis of meta-analyses (meta-epidemiologic study), but this would mainly inform on the meta-analytic process, rather than on the intervention or effect size of interest. Accordingly, if you wish to simply combine studies, than you do not need a meta-meta-analysis.

4. Selective publishing is very difficult to recognize by looking at a single study. If you have many studies, you can recognize peculiar patterns (eg. p=0.049 occurring much more frequently than p=0.051). However, if you only have one study at hand, then the only hope resides in looking at the pre-specified protocol.