How to reconcile results from many incremental hypothesis tests? I'm considering a series of controlled experiments (e.g. A/B tests) measuring the performance of a system. Each time a change to the system is found significant (via statistical inference) the test system becomes the new reference for the next experiment and the performance improvement is recorded. So I have a series of positive results of the form $(\hat{\mu}_{\Delta_{i,i-1}},\sigma^2_{\Delta_{i,i-1}})$ obtained at a given significance level $\alpha$.
My goal is to give a reasonable estimate of the cumulated improvements (i.e. the difference in performance between the first reference system and the last reference system of the series).
A perfect estimate would be to do another controlled experiment between the first reference system and the last one. Estimating the variance of such an experiment would enable computing a confidence interval which would be perfect. But running this additional experiment is costly and may not be possible for practical considerations.
$$\Delta^* = E[\Delta_{n,0}]$$
A first naïve estimate $\hat{\Delta}^{\eta}$ would be to assume the experiments were independent (which is wrong) and just multiply the improvements estimated in controlled, positive experiments ($P$):
$$\hat{\Delta}^{\eta} = \prod_{i \in P}\hat{\mu}_{\Delta_i}$$
However it is known that for a given significance level $\alpha$ the false discovery rate is asymptotically $1-\alpha$. So for a large set of experiments a number of positive (and negative) results were caused by pure luck. 
If the experiments were measuring the difference with the same base system we could use the Bonferroni correction (e.g. this question) but the incremental nature of my data doesn't seem to fit this setting.
A second approach would be to estimate the variance of $\hat{\Delta}^{\eta}$ and compute a confidence interval. However the incremental nature of the experiments seem to invalidate classical estimators (e.g. bootstrap).
So I know that the naive estimate is probably wrong but I have no clues on how to take into account the significance level of each experiment in the estimation of the cumulated improvement.
 A: Let's consider this as a physical problem, which may help to clarify the question. Suppose a large number of darts are being thrown at a dart board and each time we get closer to the center of the bull's eye we record a new best result. There may be some uncertainty to our measuring string or ruler so we assign a probability of the best answer being actually closer to the bull's eye. Now we ask what is the probability that the best answer is in some way different than all the other darts sticking in the wall, or dart board and for simplicity sake let us imagine that this is only 2-dimensional and there are no darts on the floor or in someone's foot. To do that, we can find the expected value, that is, the maximum dart 2-D concentration of a best probability distribution fit to our 2-D dart distribution. From that we do a one-sample test to see if the dart closest to the bull's eye is in any way in a significantly different position than the distribution's expected value, that is, once we know well enough what that distribution shape is.
Having done that, we can then say if getting that close to the bull's eye was a fluke, or a likely event given the arrangement of darts. Notice, it matters less how exactly close to the bull's eye our best throw is than the distance between that dart and the location of the best grouping of darts. And the question we are asking has as its limiting value "How bad is my aim? Is it significantly off from the center of the bull's eye, or not?"
I think the answer is that my aim will be off if I throw enough darts to usefully measure how far off the mark I group. I do not know if that answer is useful for the intent of the question, but, it seems that that may be what was actually being asked.
To find the probability of a dart being at a particular distance from the bull's eye, find either a 3-D density function that best fits the 2-D dart density pattern, or make a 3-D empirical histogram of the density (z-axis) versus the 2-D dart pattern (x and y axes). The next step is a bit tricky for me to understand, and if I am correct we need a special type of iso-probability contour, see here for the usual kind. We need a contour that is centered on the bull's eye, so that the area between the dart and the bull's eye in the density or z-direction of a plane perpendicular to the x y axes (looks like a 2-D function plot) is propagated as a constant area about the bull's eye. That propagated equal area around the bull'e eye contour is a 2-D probability contour x y plane region of interest (ROI) that exactly includes the closest dart throw as one point on that contour. That ROI then has a 3-D histogram or density function volume that is a probability of getting a throw that "close" or "closer" to the bull's eye. Find its volume, and that is our probability. To find the difference between that and another, still closer to the bull's eye, dart, generate a new contour and compare the ROI probabilities.
Here is one very strange feature of this problem. Just because a dart is closer to the bull's eye, does not mean that it is less likely to be that distance from the bull's eye. Picture a peanut shaped iso-area, iso-contour. If our next throw is just outside the notch in the prior peanut contour and closer to the bull's eye than the last throw occurring at one end of its peanut shaped contour, then the closer dart can have a higher probability of being that close or closer than our further away last throw. That might imply a tip for dart throwers. It may be easier to get better scores by rotating our bodies to the left or right than by correcting our throws, well, maybe.
In 2-D with 1-D distances, similar rules should apply, just easier to understand because thinking in 2-D is easier than in 3-D. However, the strange behaviour does not disappear. For example, the left "iso-contour line" can still be a different distance away from the "bull's eye" (read as target) than the right "iso-contour line." 
