Let's make the problem as simple as possible. Assume two related random variables, X1 and X2. On the basis of some data we estimate their true means mu_X1 and mu_X2 by sample means muhat_X1 and muhat_X2. These estimates are unbiased.

But now let's sort our two random variables by their sample means and look at the variable with the highest sample mean. Now for this top-of-the-list random variable the sample mean is now a biased estimator of its true mean (under some reasonable assumptions, e.g. that the means of these random variables are themselves distributed in a certain way and that distribution has a mean) -- that's easy to verify by Monte-Carlo. For obviousness, take not two but a thousand random variables and make their true means similar.

The question is what is this bias and how do I analytically calculate it? I'd also appreciate some conceptual discussion on how does estimation bias arise out of sorting by estimated values.

Let's make the problem as simple as possible. Assume two related random variables, $X_1$ and $X_2$. On the basis of some data we estimate their true means $\mu_{X_1}$ and $\mu_{X_2}$ by sample means $\hat\mu_{X_1}$ and $\hat\mu_{X_2}$. These estimates are unbiased.

But now let's sort our two random variables by their sample means and look at the variable with the highest sample mean. Now for this top-of-the-list random variable the sample mean is now a biased estimator of its true mean (under some reasonable assumptions, e.g. that the means of these random variables are themselves distributed in a certain way and that distribution has a mean) -- that's easy to verify by Monte-Carlo. For obviousness, take not two but a thousand random variables and make their true means similar.

The question is what is this bias and how do I analytically calculate it? I'd also appreciate some conceptual discussion on how does estimation bias arise out of sorting by estimated values.