# Is there a bias correction for effect size in a data mining context?

Given $K$ possible 'treatments' of some kind, and independent observations of some response under those treatments, say $X_{i,k}$ for $i=1,\ldots,n_k$ and $k=1,\ldots,K$, I am faced with the classical data-mining dilemma. The task is to simultaneously:

1. Find the treatment, say $k^*$, which maximizes effect, and
2. Estimate the effect size of treatment $k^*$.

Here you can think of effect size as a measure of location, say mean. The naive approach to this problem is as follows:

1. Estimate the effect size of each treatment. For example, if the effect size is the mean, let $\bar{X_k} = (\sum_i X_{i,k}) / n_k$.
2. Pick the treatment which maximizes the estimated effect, i.e. let $k^* = \arg\max_k \bar{X_k}$.
3. Estimate the effect size of $k^*$ by $\bar{X_{k^*}}$.

The problem, of course, is that the estimate of the optimal effect size is positively biased because the choice of optimal treatment is not independent of the estimated effect sizes, which are unconditionally unbiased.

Additionally, the goals are often somewhat broader: instead of picking the treatment which maximizes effect, we are tasked with sorting the treatments by estimated effect size and estimating the effect sizes. Because of this, I am looking for some kind of function $f$ which 'unbiases' the estimated effect sizes, but preserves monotonicity. That is, $f$ takes a $K$-vector to a $K$-vector, and

1. $f$ preserves monotonicity: if $\bar{X_{i_1}} \le \bar{X_{i_2}} \le \ldots \le \bar{X_{i_K}}$ and $(Y_1,Y_2,\ldots,Y_K) = f(\bar{X_1},\bar{X_2},\ldots,\bar{X_K})$, then $Y_{i_1} \le Y_{i_2} \le \ldots \le Y_{i_K}$.
2. $f$ (approximately, possibly under weird assumptions) unbiases the effect size estimates: if $(Y_1,Y_2,\ldots,Y_K) = f(\bar{X_1},\bar{X_2},\ldots,\bar{X_K})$, and $Y_{i_1} \le Y_{i_2} \le \ldots \le Y_{i_K}$, then $Y_{i_j}$ is a (nearly) unbiased estimate of the effect size of the $j$th smallest effect treatment.

Probably such a function would have to be aware of some additional information, like the number of observations $n_k$ and some measure of the spread of the response under each treatment. Probably the monotonicity requirement is questionable if the standard errors of the effect estimates vary wildly. Sweep that under the rug if possible.

I am also interested in approaches to this problem when the response observations are not independent, rather they are paired..

-