Suppose we made $m$ tests in 2 samples from the population. For simplicity, we compared $m$ traits (i.e. bmi, temperature, height, etc) for equality in means between 2 groups of people (i.e. Germans and Americans) 2 times. In other words, we compared $m$ traits between 2 groups once and then tried to reproduce the research one more time. We fixed level of significance $\alpha$. Our test for difference in means is "quite" reliable, which means that it is parametric and complex (correction for additional factors were required) so we do not expect our model to describe the reality completely, but the 1st type error rate is not different a lot from $\alpha$.

For the first time we performed the study, $n_1$ q-values did not exceed $\alpha$. For the second time (replication study), $n_2$ q-values were less than $\alpha$.

We figured out that after replication 2 studies agreed only in $n$ tests (which means the null hypothesis was rejected for these cases).

How can we say that the agreement in $n$ tests is significantly different from random or not?

More complicated question: suppose that $n$ times the null hypothesis for the means' equality was rejected, but only for $k$ times the direction of difference in means between Germans and Americans was the same. Here it is clear how to test if our $n$ results were just a coincidence (we toss a coin with 0.5 probability and can calculate p-value from it). Having $(n-k)$ -- number of tests that passed FDR control but are for sure wrong since the direction of change is different -- how to estimate the total amount of false discoveries in $n$ pairs of tests?


Agreement is something that is evaluated between two screeners or diagnostic tools, not statistical tests whose data is more rich and of which the p-value is merely a (bad) summary statistic. Comparing the binary significance threshold is not a good way to compare studies. For one, the "significant/non-significant" designation is (offensively) imprecise and arbitrary as a way to summarize data. If such a comparison rejects a null hypothesis from two different studies, you cannot say those studies "agree": they can still be wildly different. Furthermore, one study having p<0.05 and another having p>0.05 can agree almost identically, taking for instance the first p to be 0.049 and the other to be 0.051.

If you wish to see if studies agree, you should compare the magnitude and effect sizes using 95% confidence intervals. If the confidence intervals for an estimated mean difference overlaps the other mean difference, roughly you can say that they "agree".

To your more complicated question, if you can't be convinced yet to give up these p-values, you should be doing one sided tests. If you reject a two sided hypothesis both times in this case, they do agree according to your method, although one trial says smoking is bad for you, and the other says smoking is good for you, neither say, "there is inconclusive evidence about the health impact of smoking."

Think beyond the p-value!

  • $\begingroup$ The answer is perfect and answer the question asked, thank you! However, what should I do if the statistical model does not allow me to calculate confidence interval for difference in means? I use beta binomial regression and likelihood ratio test L(with group factor as a parameter) / L(without factor). Having low p-value means that the predictor is important and typically implies the difference in means, but the direction of change have to be determined in kind of additional test... $\endgroup$ Jan 31 '17 at 15:37
  • $\begingroup$ @GermanDemidov you can still inspect 95% CIs for the model coefficient(s). Low p-value does not necessarily mean the predictor is important, but it does imply difference of means. You need to know the direction of the difference. If you can't figure that out, you ought to throw the data away for ineffective reporting! $\endgroup$
    – AdamO
    Jan 31 '17 at 16:13
  • $\begingroup$ thanks! I got it. I can find the direction (actually I have all the raw data from 2 replications, but no prior information from other studies...) just by comparing means before test, but it does not seem a good practice for me...ie, mean BMI in Germans is bigger than in Americans prior of actual test - can I use the directional variant instead of simple test for inequality? $\endgroup$ Jan 31 '17 at 16:45
  • $\begingroup$ @GermanDemidov unfortunately no. I don't know how a p-value could arise from a single covariate, but return to my example of smoking efficacy. Two populations could show a deleterious effect of smoking, but one could be more than double in magnitude (5 fold increase in lung cancer risk) whereas the other is less so (1.2 fold increase in lung cancer risk). They agree smoking is harmful, but produce wildly different estimates that evidence effect modification due to unmeasured factors. $\endgroup$
    – AdamO
    Jan 31 '17 at 21:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.