27

Thanks for all the interesting discussions! When writing that 2008 article, it took me a while to convince myself that the distribution of replication p (the p value given by an exact replication of a study, meaning a study that is exactly the same, but with a new sample) is dependent only on p given by the original study. (In the paper I assume a normally ...


21

Summary: The trick appears to be a Bayesian approach which assumes a uniform (Jeffreys) prior for the hidden parameter ($z_\mu$ in appendix B of the paper, $\theta$ here). I believe there may be a Bayesian-style approach to get the equations given in the paper's appendix B. As I understand it, the experiment boils down to a statistic $z\sim\mathrm{N}_{\...


12

Analysis Because this is a conceptual question, for simplicity let's consider the situation in which a $1-\alpha$ confidence interval $$\left[\bar x^{(1)} + Z_{\alpha/2} s^{(1)}/\sqrt{n}, \bar x^{(1)} + Z_{1-\alpha/2} s^{(1)}/\sqrt{n}\right]$$ is constructed for a mean $\mu$ using a random sample $x^{(1)}$ of size $n$ and a second random sample $x^{(2)}$ is ...


10

The p-value from each experiment should have a uniform distribution between 0 and 1 under the null hypothesis, so tests of the null hypothesis over all experiments can be based on this. Perhaps the most common test statistic is Fisher's: for p-values $p_j$ from $m$ independent experiments the negative log of each follows an exponential distribution $$-\log ...


10

The issue has been clarified by @GeoMatt22, and I've been delighted to see @GeoffCumming coming here to participate in the discussion. I am posting this answer as a further commentary. As it turns out, this discussion goes back at least to Goodman (1992) A comment on replication, P‐values and evidence and a later reply Senn (2002) Letter to the Editor. I ...


6

I'm not familiar with this particular study but am familiar with estimating the power of an area of research using a meta-analysis. Your statement that "post hoc power is always inherently associated to the achieved p-values" suggests to me that you are assuming that the post hoc power for each individual study contributing to a single meta-analysis is based ...


6

Thanks everyone for further interesting discussion. Rather than making my comments, point by point, I’ll offer some general reflections. Bayes. I have nothing at all against Bayesian approaches. From the beginning I’ve expected that a Bayesian analysis, assuming a flat or diffuse prior, would give the same or very similar prediction intervals. There is a ...


4

[Edited to fix the bug WHuber pointed out.] I altered @Whuber's R code to use the t distribution, and plot coverage as a function of sample size. The results are below. At high sample size, the results match WHuber's of course. And here is the adapted R code, run twice with alpha set to either 0.01 or 0.05. sigma <- 2 mu <- -4 alpha <- 0.01 n....


3

I agree that, retrospectively, the power of significant effects is 1 (and the power of nonsignificant effects is 0). After all, power is the probability of rejecting the null, and if you take all information into account, you know what that result is. There are a number of people who ignore only the one detail about whether they accepted or rejected, and ...


3

I agree with Alex R's comments, and I'm expanding them into a full answer. I'll be talking about "black box" models in this answer, by which I mean machine learning (ML) models whose internal implementations are either not known or not understood. Using some sort of "Auto ML" framework would produce a black box model. More generally, many people would ...


3

I guess that your logic in here is that assuming that we know that the effect was negative (or positive), let's use one-tailed test, however the assumption is wrong, because you are conducting the replication study to check if the results hold, so you cannot take the direction for granted. Also, the choice between one-tailed and two-tailed test depends on ...


2

Although it's not completely clear what "replicability in capture percentage" is intended to mean, the question itself is interesting. It appears to ask about the utility of comparing the mean effect of one experiment to the confidence interval of the mean from another experiment, and vice versa. This can be done. As the following analysis shows, At ...


2

As I understand your question, you have done 3 separate experiments under the same conditions, each time fitting a model of the change of sample weight over time at fixed temperature to obtain estimates of the model parameter values. Your question is whether you should average the parameter values among the 3 separate analyses or to average the curves first ...


1

Preamble: Models (also as constructed by Auto-ML) can be used for many aims, not just for running tests and p-values. The first issue when investigating reproducibility is to define what exactly you want to do, how you interpret your result, and what you expect to be reproduced, and all further considerations depend on that. Now let's assume you are in fact ...


1

I was kindly referred by Dan Kassler to this very useful paper, that approaches exactly this question: http://psycnet.apa.org/psycinfo/2014-21247-001/ The authors cite this paper that I also found useful: http://search.proquest.com/docview/228434917?pq-origsite=gscholar


1

As the linked post How to conduct a meta-analysis using raw data? mentions this is widely done. If there is a concern about doing this sort of sequential analysis one might look at some of the papers which have tried to put this into a sequential framework. For instance Wetterslev and colleagues in J Clin Epid here or a follow-up here and another one in IJE ...


1

One study dealing with this I came across studied the distribution of p values in published economics articles. They found that the common thresholds are usually just undercut. I.e. p-values cluster just below 5% and 10%. This is similar to what is asked for in the original question, but probably not the only/most well-known answer. Link to the article https:...


1

I would say that should be replicated studies are ones with important, counterintuitive, too good to be true or otherwise fishi results. Replications are not being done because there is no fame in it, they are harder to publish and it's harder to get money for it, because of the perceived lack of originality and added value. If you can't get your ...


1

"What conditions would have to be true for pooling data to be appropriate?" - Not having done these separate analyses, i.e. if it were a priori decided that this would be the experimental design, but even then blocking by test would be prudent because 'as much as possible' is still never the same. The latter point, accounting for test run as a random factor, ...


Only top voted, non community-wiki answers of a minimum length are eligible