Is there a bias correction for effect size in a data mining context?

Question

Given $K$ possible 'treatments' of some kind, and independent observations of some response under those treatments, say $X_{i,k}$ for $i=1,\ldots,n_k$ and $k=1,\ldots,K$, I am faced with the classical data-mining dilemma. The task is to simultaneously:

1. Find the treatment, say $k^*$, which maximizes effect, and
2. Estimate the effect size of treatment $k^*$.

Here you can think of effect size as a measure of location, say mean. The naive approach to this problem is as follows:

1. Estimate the effect size of each treatment. For example, if the effect size is the mean, let $\bar{X_k} = (\sum_i X_{i,k}) / n_k$.
2. Pick the treatment which maximizes the estimated effect, i.e. let $k^* = \arg\max_k \bar{X_k}$.
3. Estimate the effect size of $k^*$ by $\bar{X_{k^*}}$.

The problem, of course, is that the estimate of the optimal effect size is positively biased because the choice of optimal treatment is not independent of the estimated effect sizes, which are unconditionally unbiased.

Additionally, the goals are often somewhat broader: instead of picking the treatment which maximizes effect, we are tasked with sorting the treatments by estimated effect size and estimating the effect sizes. Because of this, I am looking for some kind of function $f$ which 'unbiases' the estimated effect sizes, but preserves monotonicity. That is, $f$ takes a $K$-vector to a $K$-vector, and

1. $f$ preserves monotonicity: if $\bar{X_{i_1}} \le \bar{X_{i_2}} \le \ldots \le \bar{X_{i_K}}$ and $(Y_1,Y_2,\ldots,Y_K) = f(\bar{X_1},\bar{X_2},\ldots,\bar{X_K})$, then $Y_{i_1} \le Y_{i_2} \le \ldots \le Y_{i_K}$.
2. $f$ (approximately, possibly under weird assumptions) unbiases the effect size estimates: if $(Y_1,Y_2,\ldots,Y_K) = f(\bar{X_1},\bar{X_2},\ldots,\bar{X_K})$, and $Y_{i_1} \le Y_{i_2} \le \ldots \le Y_{i_K}$, then $Y_{i_j}$ is a (nearly) unbiased estimate of the effect size of the $j$th smallest effect treatment.

Probably such a function would have to be aware of some additional information, like the number of observations $n_k$ and some measure of the spread of the response under each treatment. Probably the monotonicity requirement is questionable if the standard errors of the effect estimates vary wildly. Sweep that under the rug if possible.

I am also interested in approaches to this problem when the response observations are not independent, rather they are paired..

Answer 1

This answer may be way, way off base as I don't understand the medical context of your question and the nature of the medical test results you allude to, but it might be possible to estimate data mining bias by some sort of Monte Carlo permutation of your results. The type of approach I'm thinking of is taken from a book, Evidence Based Technical Analysis, which has a companion website here.

Although this is written with the financial markets in mind it is, in my opinion, a book about Monte Carlo techniques as much as it is about markets.

Answer 2

Ok, duh, one approach would be to use the James-Stein shrinkage. This will not, I believe, unbias the estimates, but will reduce the mean squared error.