Are Bayes estimators immune to selection bias?

Most papers that discuss estimation in high dimension, e.g., whole genome sequence data, will often raise the issue of selection bias. Selection bias arises from the fact that, though we have thousands of potential predictors only few will be selected and inference is done on the selected few. So the process goes in two steps: (1) select a subset of predictors (2) perform inference on the select sets, e.g., estimate odds ratios. Dawid in his 1994 paradox paper focused on unbiased estimators and Bayes estimators. He simplifies the problem to selecting the largest effect, which could be a treatment effect. Then he says, unbiased estimators are affected by selection bias. He used the example: assume $$ Z_i\sim N(\delta_i,1),\quad i=1,\ldots,N $$ then each $Z_i$ is unbiased for $\delta_i$. Let $\mathbf{Z}=(Z_1,Z_2,\ldots,Z_N)^T$, the estimator $$ \gamma_1(\mathbf{Z})=\max\{Z_1,Z_2,\ldots,Z_N\} $$ is however biased (positively) for $\max\{\delta_1,\delta_2,\ldots,\delta_N\}$. This statement can be easily proven with Jensen's inequality. Therefore, if we knew $i_{\max}$, the index of the largest $\delta_i$, we will just use $Z_{i_{\max}}$ as its estimator which is unbiased. But because we do not know this, we use $\gamma_1(\mathbf{Z})$ instead which becomes biased (positively).

enter image description here

But the worrying statement Dawid, Efron and other authors make is Bayes estimators are immune to selection bias. If I will now put prior on $\delta_i$, say $\delta_i\sim g(.)$, then the Bayes estimator of $\delta_i$ is given by $$ \text{E}\{\delta_i\mid Z_i\}=z_i+\frac{d}{dz_i}m(z_i) $$ where $m(z_i)=\int \varphi(z_i-\delta_i)g(\delta_i)d\delta_i$, with $\varphi(.)$ the standard Gaussian.

If we define the new estimator of $\delta_{i_{\max}}$ as $$ \gamma_2(\mathbf{Z})=\max\{\text{E}\{\delta_1\mid Z_1\},\text{E}\{\delta_2\mid Z_2\},\ldots,\text{E}\{\delta_N\mid Z_N\}\}, $$ whatever $i$ you select to estimate $\delta_{i_{\max}}$ with $\gamma_1(\mathbf{Z})$, will be the same $i$ if the selection was based on $\gamma_2(\mathbf{Z})$.This follows because $\gamma_2(\mathbf{Z})$ is monotone in $Z_i$. We also know that $\text{E}\{\delta_i\mid Z_i\}$ shrinkes $Z_i$ towards zero with the term, $\frac{d}{dz_i}m(z_i)$ which reduces some of the positive bias in $Z_i$. But how do we conclude that Bayes estimators are immune to selection bias. I really don't get it.

  • 1
    $\begingroup$ Given that you are referring a claim in a piece of literature, can you please give a full situation and page reference, so that we can read the full context of this claim. $\endgroup$
    – Ben
    Mar 14 '18 at 2:10
  • $\begingroup$ Is defining an estimator as the max of Bayes estimators still a Bayes estimator? $\endgroup$
    – Xi'an
    Mar 14 '18 at 12:34
  • $\begingroup$ Example 1 in the paper. $\endgroup$ Mar 14 '18 at 12:55

As described above, the issue stands with drawing inference on the index and value, (i⁰,μ⁰), of the largest mean of a sample of Normal rvs. What I find surprising in Dawid's presentation is that the Bayesian analysis does not sound that much Bayesian. If given the whole sample, a Bayesian approach should produce a posterior distribution on (i⁰,μ⁰), rather than follow estimation steps, from estimating i⁰ to estimating the associated mean. And if needed, estimators should come from the definition of a particular loss function. When, instead, given the largest point in the sample, and only that point, its distribution changes, so I am fairly bemused by the statement that no adjustment is needed.

The prior modelling is also rather surprising in that the priors on the means should be joint rather than a product of independent Normals, since these means are compared and hence comparable. For instance a hierarchical prior seems more appropriate, with location and scale to be estimated from the whole data. Creating a connection between the means... A relevant objection to the use of independent improper priors is that the maximum mean μ⁰ then does not have a well-defined measure. However, I do not think a criticism of some priors versus other is a relevant attack on this "paradox".

  • 1
    $\begingroup$ Seems to me that all the protection needed should be coded in the prior that connects all the unknown means. If the prior makes large differences between means very unlikely, that will be reflected in the posterior making it perfect. $\endgroup$ Mar 15 '18 at 12:52
  • $\begingroup$ @Xi'an can you give an example of how you will place a prior on $(i,\mu)$? $\endgroup$ Mar 15 '18 at 13:48
  • $\begingroup$ @Frank Harrel, consider for example $\delta_i \sim N(a,1)$ and $Z_i\sim N(\delta_i,1)$. The unbiased estimator of $\delta_i$ is $Z_i$. The Bayes estimator of $\delta_i$ is $E(\delta_i|Z_i)$. The if $Z_{i^0}$ is the largest $Z_i$ so is $E(\delta_{i^0}|Z_{i^0})$, because the Bayes estimator is monotone in $Z_i$. No matter how informative the prior is, this will not change. However, $E(\delta_{i^0}|Z_{i^0})$ reduces the positive Bayes in $Z_{i^0}$. But if the wrong $i^0$ was chosen the Bayes estimator cannot correct this. $\endgroup$ Mar 15 '18 at 13:58
  • $\begingroup$ @ChamberlainFoncha: The Bayes estimator is only $\mathbb{E}[\delta_i|Z_i]$ when the $\delta_i$'s are a priori independent. A joint prior on $i$ and the $\mu_i$'s makes them dependent actually. $\endgroup$
    – Xi'an
    Mar 15 '18 at 14:14
  • $\begingroup$ And any prior is acceptable from a Bayesian viewpoint, for instance a uniform distribution on the index and a hierarchical prior on the $\mu_i$'s. $\endgroup$
    – Xi'an
    Mar 15 '18 at 14:33

Even if a bit counter-intuitive the statement is correct. Assume $i^*=5$ for this experiment, then the posterior for $\mu_5$ is really $N(x_5,\sigma^2)$. This counter-intuitive fact is a bit similar to Bayes being immune to (secret) early stopping (that is also very counter-intuitive).

The Bayesian reasoning would lead to false conclusions if for each such experiment (imagine your repeat it a few times), only the results for the best variety would be kept. There would be data selection and Bayesian methods are clearly not immune to (secret) data selection. Actually no statistical method is immune to data selection.

If such a selection was done, a complete Bayesian reasoning taking this selection into account would easily correct the illusion.

However the sentence "Bayes estimator are immune to selection Bias" is a bit dangerous. It is easy to imagine situations where "selection" means something else, like for example selection of explanatory variables, or selection of data. Bayes is not clearly immune to this.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.