replicability and statistical significance

Data for two nearly identical clinical trials. Study A had 750 pts, 374 placebo and 376 drug. Study B had 752 pts, 376 placebo and 376 drug. Same protocol for both. Study A showed mean change from baseline of -0.17 indicating improvement in sign, p<0.05 and 95% CI <0 while Study B showed mean change from baseline of -0.03, p>0.05 and 95% CI >0. 95%. CIs overlap for both studies and mean for Study A is within the CI of Study B and vice versa. Each Study has a capture percentage of 83.4% for the mean of the other study. It seems the probability that neither study captures the mean of the other is the product of each individual failure to capture the mean of the other study... i.e., (1.00-0.834) X (1.00-0.834)= (0.166) X (0.166)= 0.0276. Might this demonstrate statistically significant replicability in capture percentage despite failure of one study to reach p<0.05?

• If studies A and B would have st. error of the mean of e.g. 0.07, then A would be significant (0.17 > 2*0.07), B would be non-significant (0.03 < 2*0.07), but A and B would not be significantly different (0.17-0.03 < 2*0.07*sqrt(2)). That's a normal situation. – amoeba Sep 18 '17 at 14:35

Although it's not completely clear what "replicability in capture percentage" is intended to mean, the question itself is interesting. It appears to ask about the utility of comparing the mean effect of one experiment to the confidence interval of the mean from another experiment, and vice versa. This can be done.

As the following analysis shows,

1. At least approximately, the chance that neither experiment's confidence interval includes the other's mean--the chance of "overlap"--does not depend on the experimental outcomes.

Conceptually, this chance depends only on the true underlying distribution: it cannot possibly depend on the outcomes. This is analogous to asserting that the chance of a coin landing heads does not depend on the result of just one or two flips (or even any finite number of flips). The coin's chance is what it is; the flips merely reflect, quite indirectly, that underlying chance.

2. The chance of non-overlap is appreciably large.

In the situation posited in the question, the chance of non-overlap when both experiments are run at a size $\alpha=0.05$ (that is, 95% confidence) is approximately

$$2\Phi(-Z/\sqrt{2}) = 2\Phi(-1.96/ \sqrt{2}) \approx 16.6\%$$

where $1.96 = Z_{1-0.05/2}$ is used for computing two-sided $95\%$ confidence intervals.

Finally, since this chance says nothing whatsoever about the true effect size, studying overlap is really a way of assessing whether the experiments concern the same underlying distribution or not.

A closely related analysis (comparing confidence intervals) appeared earlier at https://stats.stackexchange.com/a/18259/919. It examined the case where two confidence intervals do not intersect at all.

To get a sense of what non-overlap might tell us, let's adopt assumptions that--although not entirely realistic--are close enough to how many experiments work that the results will be informative. Namely, suppose that experiment $i\in\{1,2\}$ collects a sample of size $c_i$ from a distribution $F$ for its control group and a sample of size $n_i$ from a distribution $G$ for its treatment group. Let $\sigma_F^2$ be the variance of $F$ and $\sigma_G^2$ that of $G$.

The variances of the experimental effects $m_i$ (the differences between treatment and control group means) therefore are

$$\operatorname{Var}(m_i) = s_i^2 = \frac{\sigma_F^2}{c_i} + \frac{\sigma_G^2}{n_i}.$$

Let's assume the $c_i$ and $n_i$ are large enough that the sample distributions of the means can be approximated by Normal distributions.

Finally, let us suppose $\sigma_F^2$ and $\sigma_G^2$ are known (so we can simplify the analysis while still capturing its essence).

A symmetric two-sided confidence interval for the mean effect in experiment $i$ will therefore have endpoints close to $$m_i\pm Z s_i$$ where $Z$ is a number depending on the confidence level. The event that each CI fails to cover the other experiment's mean is equivalent to the difference in the means exceeding the larger of the interval radii, $Z\max{(s_1,s_2)}$; namely

$$\mathcal{E}:m_2 - m_1 \notin [-Z\max{(s_1,s_2)},Z\max{(s_1,s_2)}].$$

We can compute this because the random variable $m_2-m_1$ is Normal with mean $0$ and (since the experiments are independent) its variance is $s_1^2 + s_2^2$. Writing $\Phi$ for the standard Normal distribution, this chance is

$$\Pr(\mathcal{E}) = 2\Phi\left(-Z\frac{\max{(s_1,s_2)}}{\sqrt{s_1^2 + s_2^2}}\right).$$

When the experiments use groups of comparable sizes, as in the question, $s_1 \approx s_2$, and therefore we may approximate the ratio easily by writing both variances as $s$ and computing

$$\frac{\max{(s_1,s_2)}}{\sqrt{s_1^2 + s_2^2}} \approx \frac{s}{\sqrt{s^2+s^2}}=\sqrt{1/2}\approx 0.7.$$

Therefore

$$\Pr(\mathcal{E}) \approx 2\Phi\left(-Z/\sqrt{2}\right).$$

Since $Z$ is a value for which $\Phi(-Z)\approx \alpha/2$ and $0 \lt Z/\sqrt{2} \lt Z$, the non-overlap probability will always be greater than the test size $\alpha$.

I think "statistically significant replicability" is a rather strange idea. With statistical significance, the null hypothesis is usually that there is nothing going on and you usually want to reject that null. Here, you would be happier if the null was true (that is, the improvement is the same in the two trials). If you want a statistical test of this, look into tests of equivalence.

But, really, I think the important bit is not statistical significance but, rather, whether -0.17 is meaningfully different from -0.03. We have no way of knowing that - it depends on what is being measured.

• Apologies on error – Tom G Sep 18 '17 at 3:19
• Apologies on error. I will look into tests of equivalence. There is no statistically significant (two-sided alpha < 0.05) difference in the mean change from baseline for Study A (-0.17) and Study B (-0.03). – Tom G Sep 18 '17 at 3:22