Selection bias in variable selection

Question

Let $$ \theta_i\sim N(0,1), \quad i=1,\ldots,N $$

and let $$ \theta_{i_{\max}}=\max{\theta_i,\quad i=1,\ldots,N}. $$ Assume that we now observe data independently from $$ Z_i\sim N(\theta_i,1). $$ Let $$ Z_{\max}=\max Z_,i=1,\ldots,N. $$ I leave $\theta_i$s fixed and resample $Z_i$s from their distributions 1000 times. I create the figure below.

What do we learn from the figure?

1. $Z_{i_{\max}}$ is unbiased for $\theta_{i_{\max}}$, this not suprising since each $Z_i$ is unbiased for $\theta_{i}$ for all $i$.

2. $Z_{\max}$ is postively biased for $\theta_{i_{\max}}$.

The only reason $Z_{{\max}}$ is biased for $\theta_{i_{\max}}$ because we lack knowledge of $i_{\max}$. This type of bias is called selection bias. Immediately we know $i_{\max}$, $Z_{i_{\max}}$ becomes unbiased as we see in the figure. Bayesians claim that the extra information we lack that brings bias is in the prior. So that the resulting Bayes estimator with respect to the squared loss has a reduced selection bias. This is indeed true, the Bayes estimator will be $E\{\theta_i|Z_i\}=Z_i/2$. This Bayes estimator has selection bias half the size of that of $Z_i$, though it is biased for $\theta_i$.

Everthing uptill now looks good and I am happy. Where do things start going bad? Now I need to identify $i_{\max}$. Say the indices refers to different treatments, I want the treatment with the largest effect. So that not just their effect sizes are important to me I need to know the treatment with the largest effect.

Now define $I$ as the index of $Z_{\max}$, then $I$ is a random variable since it will change from sample to sample. If I choose to define $I$ based on the Bayes estimator instead i.e., $I_B$ is the index of $\max Z_i/2$, then $I=I_B$. So the Bayesian prior we brought in cannot help us in improving the estimator $I$. How can I have a Bayes estimator of $i_{\max}$? Or how can I improve the estimator $I$?

Answer 1

The structure of your problem implies that all $i$ are equally likely to be $i_{max}$; given that, and the fact that your $Z_i$ are i.i.d., it is indeed the case that the Bayes estimator of $i_{max}$ is the index of $Z_{max}$. Another way of putting it is that, with respect to $i_{max}$, your prior information is uninformative, so $I_B = I$.

Unfortunately, there is no way to improve this estimator, based on the information in this question. If, on the other hand, we allowed, for example, different variances for the different distributions indexed by $i$, or different sample sizes from each distribution, or different prior information about the parameters of the various distributions, then something could be done - what, of course, depending upon the nature of the altered assumptions / prior information!