# Multiple comparisons correction for dependent comparisons

In this blog post the authors discuss simultaneously estimating quantiles, and constructing a simultaneous confidence envelope for the estimation which covers the whole quantile function. They do this by bootstrapping and then computing pointwise bootstrap confidence intervals and applying a Bonferroni type correction for multiple comparisons. Since the comparisons are not independent, they compute something like an effective number of independent trials according to a formula

$$N_{eq}=\frac{N^2}{\sum_{i,j}r(b_i,b_j)}$$

where $$N$$ is the number of points to be estimated and $$r(b_i,b_j)$$ is the sample correlation between the $$ith$$ and $$j$$th bootstrap vectors.

My question is where this formula comes from. They provide a link to a source, but I don't see this formula in the source. Is anyone aware of this particular correction being used in the literature?

Let's consider a simpler, but closely related, problem. Assume you have a vector $$x$$ whose entries have a correlation matrix $$R$$. If $$R$$ is diagonal, then the variance of the sample mean of $$x$$ is $$\sigma^2_{\bar{x}} = \sum \sigma^2_i/n^2$$. If all the $$\sigma^2_i$$ are equal, this reduces to the more commonly seen $$\sigma^2/n$$, and we can rearrange terms to get the sample size as a function of the two variances:

$$n = \sigma^2 / \sigma^2_{\bar{x}}$$

If, however, $$R$$ is not diagonal, then the variance of $$\bar{x}$$ is equal to:

$$\sigma^2_{\bar{x}} = \sum_i\sum_j\sigma_i\sigma_jr_{ij}/n^2$$

In the case where $$\sigma_i = \sigma_j \space \forall \space i,j$$, this reduces to:

$$\sigma^2_{\bar{x}} = \sigma^2 \sum_i\sum_jr_{ij}/n^2$$

By analogy with the expression for the sample size in the uncorrelated case, the ratio between this and $$\sigma^2$$ is the "effective sample size" $$n_{ess}$$:

$$n_{ess} = \sigma^2 / \sigma^2_{\bar{x}} = {\sigma^2 \over \sigma^2\sum_i\sum_jr_{ij}/n^2} = {n^2 \over \sum_i\sum_jr_{ij}}$$

If $$R$$ is diagonal, then $$\sum_i\sum_jr_{ij} = n$$, as the $$n$$ diagonal elements of $$R$$ equal $$1$$ and all the off-diagonal elements equal $$0$$, and the effective sample size equals the actual sample size, as we would expect.

Comparing the effective sample size to the actual sample size provides an estimate of the impact of nonzero correlation on your estimation problem. Clearly, if your correlations are high (either positive or negative), or you have a lot of them, the effects can be pretty substantial.

The relationship between this example and your case is as follows. Instead of "sample size", consider $$n$$ as the number of quantiles being estimated. The basic Bonferroni correction adjusts for $$n$$ comparisons. Because the estimates of quantiles are known to be correlated, however, the Bonferroni correction will (in the case of positive correlations) be too pessimistic (even for Bonferroni, which in general is quite pessimistic to begin with).

To see this, consider a situation where you are estimating ten quantiles and the errors in the quantile estimates are perfectly correlated. For example, assume you are estimating the quantiles of a Normal distribution with known variance equal to one; your estimates will all be of the form $$\bar{x} \pm k$$, where $$k$$ depends upon the quantile in question. The errors in your quantile estimates will all be exactly equal to the error in your estimate of the mean, so the collection of pointwise confidence intervals will equal the simultaneous confidence interval. In effect, you have just one estimate, but Bonferroni (without correction) will adjust your confidence intervals as though you had ten estimates, making it wider than it ought to be. The above calculation, with $$r_{ij}=1 \space \forall \space i,j$$, will result in $$n_{ess} = 1$$, as it should, and the Bonferroni interval using the effective sample size will, in this case, be correct.

As for the source, it's in the paper (Solow-Polasky) linked to in the link you provide; the formula is equation (7), although the notation is completely different.