# Selection bias in variable selection

Let $$\theta_i\sim N(0,1), \quad i=1,\ldots,N$$

and let $$\theta_{i_{\max}}=\max{\theta_i,\quad i=1,\ldots,N}.$$ Assume that we now observe data independently from $$Z_i\sim N(\theta_i,1).$$ Let $$Z_{\max}=\max Z_,i=1,\ldots,N.$$ I leave $\theta_i$s fixed and resample $Z_i$s from their distributions 1000 times. I create the figure below.

What do we learn from the figure?

1. $Z_{i_{\max}}$ is unbiased for $\theta_{i_{\max}}$, this not suprising since each $Z_i$ is unbiased for $\theta_{i}$ for all $i$.

2. $Z_{\max}$ is postively biased for $\theta_{i_{\max}}$.

The only reason $Z_{{\max}}$ is biased for $\theta_{i_{\max}}$ because we lack knowledge of $i_{\max}$. This type of bias is called selection bias. Immediately we know $i_{\max}$, $Z_{i_{\max}}$ becomes unbiased as we see in the figure. Bayesians claim that the extra information we lack that brings bias is in the prior. So that the resulting Bayes estimator with respect to the squared loss has a reduced selection bias. This is indeed true, the Bayes estimator will be $E\{\theta_i|Z_i\}=Z_i/2$. This Bayes estimator has selection bias half the size of that of $Z_i$, though it is biased for $\theta_i$.

Everthing uptill now looks good and I am happy. Where do things start going bad? Now I need to identify $i_{\max}$. Say the indices refers to different treatments, I want the treatment with the largest effect. So that not just their effect sizes are important to me I need to know the treatment with the largest effect.

Now define $I$ as the index of $Z_{\max}$, then $I$ is a random variable since it will change from sample to sample. If I choose to define $I$ based on the Bayes estimator instead i.e., $I_B$ is the index of $\max Z_i/2$, then $I=I_B$. So the Bayesian prior we brought in cannot help us in improving the estimator $I$. How can I have a Bayes estimator of $i_{\max}$? Or how can I improve the estimator $I$?

The structure of your problem implies that all $i$ are equally likely to be $i_{max}$; given that, and the fact that your $Z_i$ are i.i.d., it is indeed the case that the Bayes estimator of $i_{max}$ is the index of $Z_{max}$. Another way of putting it is that, with respect to $i_{max}$, your prior information is uninformative, so $I_B = I$.

Unfortunately, there is no way to improve this estimator, based on the information in this question. If, on the other hand, we allowed, for example, different variances for the different distributions indexed by $i$, or different sample sizes from each distribution, or different prior information about the parameters of the various distributions, then something could be done - what, of course, depending upon the nature of the altered assumptions / prior information!

• thanks a lot for your clear answer. If I will now say a certain fraction perhaps $\kappa$ of $\theta_i\sim N(1,1)$ and the other fraction are $\theta_i\sim N(0,1)$, the $i$s do not have a uniform distribution anymore. What is the distribution of the $i$s now? Mar 20, 2018 at 19:38
• After rethinking the problem I disagree with your view of the problem. we quantify uncertainty of $\theta_i$ with $N(0,1)$ and not that $\theta_i$ is a random variable. So the type of uncertainty we put on $i$ can't be uniform. Mar 20, 2018 at 21:12
• $i$ refers to whichever $\theta_i$ is largest; since the probability, subjective or otherwise, that $\theta_i$ is the largest equals the probability that $\theta_j$ is the largest for all $i,j$, it must be that, implicitly, the probability, subjective or otherwise, that any given $i$ corresponds to the largest $\theta_i$ is uniform. Mar 20, 2018 at 22:27
• I will demonstrate to you that this is not the case, and if you want we can have a private chat. Lets take $N=2$, then $\Pr(I=1)=\Phi\left(\frac{\theta_1-\theta_2}{\sqrt{2}}\right)$, $I$ has a Bernoulli distribution. So that the distribution of $I$ only dependence on the difference $\theta_1-\theta_2$. Therefore any prior info that will help improve $I$ should be put on the difference $\theta_1-\theta_2$. Clearly $\theta_1-\theta_2\sim N(0,2)$. What do you think of this line of reasoning? Mar 21, 2018 at 9:30
• I'm not seeing how that shows that anything I wrote is not the case... Mar 21, 2018 at 13:08