Simple questions about reproducibility Suppose we made $m$ tests in 2 samples from the population. For simplicity, we compared $m$ traits (i.e. bmi, temperature, height, etc) for equality in means between 2 groups of people (i.e. Germans and Americans) 2 times. In other words, we compared $m$ traits between 2 groups once and then tried to reproduce the research one more time. We fixed level of significance $\alpha$. Our test for difference in means is "quite" reliable, which means that it is parametric and complex (correction for additional factors were required) so we do not expect our model to describe the reality completely, but the 1st type error rate is not different a lot from $\alpha$.
For the first time we performed the study, $n_1$ q-values did not exceed $\alpha$. For the second time (replication study), $n_2$ q-values were less than $\alpha$.
We figured out that after replication 2 studies agreed only in $n$ tests (which means the null hypothesis was rejected for these cases).
How can we say that the agreement in $n$ tests is significantly different from random or not?
More complicated question: suppose that $n$ times the null hypothesis for the means' equality was rejected, but only for $k$ times the direction of difference in means between Germans and Americans was the same. Here it is clear how to test if our $n$ results were just a coincidence (we toss a coin with 0.5 probability and can calculate p-value from it). Having $(n-k)$ -- number of tests that passed FDR control but are for sure wrong since the direction of change is different -- how to estimate the total amount of false discoveries in $n$ pairs of tests? 
 A: Agreement is something that is evaluated between two screeners or diagnostic tools, not statistical tests whose data is more rich and of which the p-value is merely a (bad) summary statistic. Comparing the binary significance threshold is not a good way to compare studies. For one, the "significant/non-significant" designation is (offensively) imprecise and arbitrary as a way to summarize data. If such a comparison rejects a null hypothesis from two different studies, you cannot say those studies "agree": they can still be wildly different. Furthermore, one study having p<0.05 and another having p>0.05 can agree almost identically, taking for instance the first p to be 0.049 and the other to be 0.051.
If you wish to see if studies agree, you should compare the magnitude and effect sizes using 95% confidence intervals. If the confidence intervals for an estimated mean difference overlaps the other mean difference, roughly you can say that they "agree".
To your more complicated question, if you can't be convinced yet to give up these p-values, you should be doing one sided tests. If you reject a two sided hypothesis both times in this case, they do agree according to your method, although one trial says smoking is bad for you, and the other says smoking is good for you, neither say, "there is inconclusive evidence about the health impact of smoking."
Think beyond the p-value!
