Could we estimate replicability of empirical research with conformal predictions?

Question

A review article Threats of a Replication Crisis in Empirical Computer Science reviews reproducibility issues. The authors present distinctions among repeatability, replicability, and reproducibility. They were a bit pessimistic. Now, the question is if there is a statistically "robust" technique or framework that actually provides confidence on repeatability? For example, how can we report a probability or an index that the result of randomised control trials are repeatable: given a significance level and the effect-size? Informally, we would report, $(\mathbb{1}_{s},\epsilon, \rho)$, where $\mathbb{1}_{s}$ is the significance level, effect size $\epsilon$ and $\rho$ is the confidence on both statistical significance and the effect size. Maybe this can be achieved with conformal predictions.

Note: The question is actually about if we can use conformal predictions in getting estimate for repeatability.

Answer 1

Yes, we can absolutely construct predictive p-values and prediction intervals aimed at covering the observed result of a repeated experiment. However, I do not think this will solve what many call the "replication/reproducibility crisis." If an experimental result is indeed an exceedingly rare event, not only might we incorrectly reject a hypothesis about a population-level parameter, we might also incorrectly reject a hypothesis about a future experimental result.

If we are always vigilant to remember that no hypothesis is proven false with a single small p-value, nor is it proven true with a large one, then there is no replicability crisis. We need to temper our expectations when it comes to interpreting statistical results.

Under a Neyman-Pearson framework we have inductive behavior based on controlling a long-run error rate. Under a Fisherian framework we have inductive reasoning based on an evidential p-value providing the weight of the evidence. In either approach if we "rule out" a hypothesis only one of two things can be true: either the hypothesis is true and we have witnessed an exceedingly rare event, or the hypothesis is false. These statements are falsifiable through repeated sampling. The only irrefutable evidence is to actually repeat an experiment many times to see if the results are replicable/reproducible. More emphasis should be placed on confidence intervals and prediction intervals - the set of all hypotheses that do well at explaining the observed data. Additionally, we should examine both meta-analytic (conformal) approaches as well as multiplicity-adjustments.

The role of journal publications is to highlight the work of researchers that is worthy of attempted replication. Nothing more. A single failed attempt at replicating a small p-value is not irrefutable evidence that the conclusions of the original experiment represent a false positive (there are many hypotheses in an observed confidence interval that do well at explaining the observed result).

Answer 2

There are a couple of drivers to any replication crisis in any field that uses statistical methods.

The first is the p-value problem. As a device of rhetoric, a stand-alone p-value is a wonderful tool. It does not depend on the frame of reference of any observer. It serves as a mutually agreed upon convention. Much of the crisis with COVID from the anti-vax community is the rejection of the conventions. In that case, a p-value becomes a club similar to a biblical verse in bible-thumping. Conversely, the radical embrace of the convention also causes problems because p-values rarely stand alone. At the extreme, you get data mining or replication crises.

If journals only publish articles where there is a new finding, then p-values are valid in the sense intended only if they are not published. If $H_0$ is true and $\alpha=.05$, then it would be expected that up to five percent of studies would falsify the null. That would imply that 100% of studies under that rule of publication will be false reports, because $H_0$ is actually true.

On the flip side, if $H_0$ is false, the percentage of false acceptances will, on average, depend on sample and effect sizes.

That leaves us with a mixture of true and false articles. That is quite a mess. It is also the seeding for a replication crisis as it would appear journal publication defeats the purposes of journal publication.

Of course, that is a shallow view, somewhat. A sophisticated scientist is fully conscious of the issues in the field and evaluates articles carefully.

Still, it would be better to not have the problem or to make it smaller.

There are a number of solutions to this type of crisis.

The first is for journals to accept articles for publication prior to the experiment. Require that the proposal for the research be made to the editor and that all results, good and bad, get published. That forces an extensive prior review and confidence by the editor of your writing skills.

The second is for a field to have a well subscribed to Journal of Non-significant Findings. A p-value with 1000 experiments and 50 false positives will not be a problem if there are 950 articles with no significant findings.

Project DARE is an example of a false finding. Lots of money goes into DARE every year, but a meta-analysis of three dozen studies shows it has no impact at all. The single best drug and alcohol prevention program in the United States is the visiting nurses program for expecting and new mothers. The children of those mothers have been shown to use 50-75% less alcohol than their peers at age 16.

False positives are a real problem because it causes the misallocation of funds into non-productive projects.

One other suggestion is in the literature and that is switching to Bayesian probability. There is some value there, but I do not think as much as is advertised.

The problem is in setting a publication standard. In a review, Wetzels et al. in Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t Tests in 70% of the t-tests reporting p-values between .05 and .01, the Bayesian support was only anecdotal. Although Bayesian methods do not have a concept equivalent to "significance", to say support is only anecdotal is less than what a psychologist might mean when they say a finding is statistically significant.

Also, as a side note, some studies that found against the null using a Frequentist standard found for the null under a Bayesian standard.

Switching to a Bayesian standard likely will not impact the discussion unless the idea of a null hypothesis is discarded completely.

Bayesian methods do not have null hypotheses. In Wetzel's paper, they accept the region of the published null hypothesis, in essence, as hypothesis one. Conversely, the region of the alternative would be hypothesis two. There is no restriction on the number of hypotheses that you can have in a Bayesian method.

Nonetheless, this creates a new class of problems in computer science. Bayesian hypotheses have to be mutually exclusive and exhaustive. Null hypothesis testing is always exhaustive. The null is true or it is not.

A Bayesian article testing the mutually exclusive and believed to be an exhaustive set of hypotheses $H_1\dots{H}_{10}$ will be a problem if a new discovery changes a technology or computer architecture so that there is also a hidden $H_{11}$. For example, think of what a test would have been like in the 1970s. Imagine a research result that is true in the '70's but not true under quantum computing.

You still have the editor's problem of "how uninteresting that finding was" that would be equivalent to not falsifying the null.

One other possible solution would require funding. That would be a funded system where graduate students systematically reproduce the findings in journal articles and publish in a Journal of Replication Studies in Computer Science. The difficulty is that you must not stop at one replication study.

Study one found a significant finding. Study two did not. Why would it be the case that it was the first study that had a non-representative sample?

Just a note, that is also the reason a simple prediction study won't work. What if it is the prediction study with the non-representative sample?

In the best of all worlds, each field would have journals with ex ante approval, journals with post hoc non-significant findings, a journal of replications, and a systematic journal of Bayesian criticism of Frequentist studies.