Combining data sets that have been adjusted for multiple testing I have what I think is a simple question. 
Let say I have 5 data sets with with a variable number of tests (say 1-10K) and I control the FDR  in each set to 1% using BH. If I combine these BH adjusted sets into one big set, will the FDR of the big set still be 1%? Each data set was analyzed identically, but each set has a different number of total tests. If I combined all the data first and controlled the FDR using BH on the entire dataset all at once, would the results be identical to the method where I combined them from 5 separate analysis?
Something tells me no, but I don't know why
 A: The false discovery rate is describes the probability of making a Type I error, conditional on some rejections of the null hypothesis (i.e. "discoveries") being true. This rate is scalable, meaning that the rejection probability holds even if we change the size of number of comparisons (say in aggregating the results from different studies).
So if you have, say, X number of rejected null hypotheses using the FDR out of 100 total null hypotheses, if you add 100 more null hypotheses, then controlling the FDR at the same level will insure that those X rejected null hypotheses still have the same probability of rejection under the new family of comparisons. That is not necessarily the case with family-wise error rate adjustments.
A: I disagree with the accepted answer, and even setting aside the portion I disagree with, I also think it's incomplete.
Here's what's missing. It matters a lot whether there is overlap in the set of hypotheses tested from one dataset to the next. The nightmare scenario is that hypotheses 1-100 are correctly rejected in both input sets, then hypotheses 101 and 102 are incorrectly rejected, one in each input set. This doubles your FDR in the union discovery set. The rules given in this paper  may be helpful to compensate for this phenomenon. Journal site
The case with no overlap in hypotheses is simpler, because this can't happen.
For the case with no overlap, consider merging two disjoint sets of discoveries. Let $a$ and $b$ be the number of false discoveries in the first and 2nd set respectively. Let $c$ and $d$ be the total number of discoveries. Let $\lambda = \frac{c}{c+d}$. I'm a little unclear on terminology but I'll write "FDP" for the false discovery proportion observed in the data (with the convention FDP=0 for cases with no discoveries), and I'll write "FDR" for $E[FDP]$. Then the combined FDP is a convex combination of the individual FDP's.
$$\begin{align}
\frac{a + b}{c + d}
&= \frac{a}{c + d} + \frac{b}{c + d} \\
&= \frac{c}{c+d}\frac{a}{c} + \frac{d}{c+d}\frac{b}{d} \\
&= \lambda\frac{a}{c} + (1-\lambda)\frac{b}{d} \\
\end{align}$$
It's tempting to conclude $$E[\frac{a + b}{c + d}] = \lambda E[\frac{a}{c}] + (1-\lambda)E[\frac{b}{d}] = \alpha$$ where $\alpha$ is the FDR of the two sets you started with, but since $\lambda$ is random and not independent of abcd, it cannot be simply passed through the expectation like that. Yet, when I simulate this process, it seems to work just fine. The combined FDP hovers around the nominal FDR for the sets of discoveries used as input. This answer says much the same: good behavior in practice, but nobody can quite prove it in theory.
Here's the simulation code.
do_one = function(n){
  means = rep(1, 100*10^n) %>% c(rep(0, 900*10^n))
  datasets = lapply(means, function(m) rnorm(mean = m, n = 10))
  tests = lapply(datasets, t.test)
  p = sapply(tests, magrittr::extract2, "p.value")
  q = p.adjust(p, method = "fdr")
  return(
    c( 
      n_true_discoveries = sum(q[1:(100*10^n)]<0.5),
      n_false_discoveries = sum(q[-(1:(100*10^n))]<0.5)
    )
  )
}

# Get TP and FP counts at FDR<0.5
output = sapply(c(1,1), do_one)
# Is the combined rate <0.5?
output %>% rowSums %>% prop.table


Here's the part of the accepted answer that I disagree with.

So if you have, say, X number of rejected null hypotheses using the FDR out of 100 total null hypotheses, if you add 100 more null hypotheses, then controlling the FDR at the same level will insure that those X rejected null hypotheses are still rejected under the new family of comparisons. That is not necessarily the case with family-wise error rate adjustments.

Here's some R code providing a counterexample to this claim. This code tests some null hypotheses while controlling the FDR, then adds more null hypotheses and shows that the same ones from before can no longer be rejected. Thus, use of FDR control in place of FWER control still does not completely relieve the burden of multiple testing.
# Test 1000 independent hypotheses
set.seed(10)
means = rep(1, 100) %>% c(rep(0, 900))
datasets = lapply(means, function(m) rnorm(mean = m, n = 10))
tests = lapply(datasets, t.test)
p = sapply(tests, magrittr::extract2, "p.value")
q = p.adjust(p, method = "fdr")
sum(q<0.1)

# Now add more null hypotheses.
more_means = c(rep(0, 10000))
more_datasets = lapply(more_means, function(m) rnorm(mean = m, n = 10))
more_tests = lapply(more_datasets, t.test)
more_p = sapply(more_tests, magrittr::extract2, "p.value")
more_q = p.adjust(c(p, more_p), method = "fdr")
sum(more_q<0.1)

