Admissible Empirical Bayes Examples

Question

I would like to hear about a few simple empirical bayes estimators that are admissible for high (i.e. at least 3) dimensional parameter space.

What are some textbook lollipop examples to study for beginners with easy derivations?

Answer 1

The question has no clear answer because the empirical Bayes formulation does not & cannot specify how the hyperparameter is estimated.

Take the simplest Normal mean estimation problem. When$$X\sim\mathcal N_p(\theta,I_p)\qquad\qquad\theta\sim\mathcal N_p(0,\sigma^2 I_p)$$the Bayes estimator of $\theta$ is$$\delta^\pi(x)=\frac{\sigma^2}{1+\sigma^2}x$$ If $\sigma$ is unknown, a corresponding empirical Bayes estimator is therefore$$\frac{\hat\sigma^2}{1+\hat\sigma^2}x$$where $\hat\sigma^2$ is an estimator of $\sigma^2$ based on the marginal distribution of $x$ $$m(x|\sigma)=\int f(x|\theta) \pi(\theta|\sigma)\,\text{d}\theta$$ But since there is no constraint on the choice of $\hat\sigma^2$, this estimator can be any (positive) function of $x$ and the collection of empirical Bayes estimators thus includes all shrinkage estimators and therefore all admissible generalised Bayes estimators of $\theta$ (see Strawderman and Cohen, 1971).

For instance, the admissible minimax Bayes estimators of Strawderman (1971) $$\delta_c(x)=\left[ 1 - \frac{\int_0^1 \lambda^{p/2-c+1}e^{-\lambda|x|^2}\text{d}\lambda}{\int_0^1 \lambda^{p/2-c}e^{-\lambda|x|^2}\text{d}\lambda} \right]x\qquad \text{where}\ 3-p/2\le c\le 2$$ are shrinkage estimators that can all be interpreted as empirical Bayes estimators, whatever the constant $c$ is. The estimator $\delta_c$ is furthermore a proper Bayes estimator when $c<1$ (and $p>5$) and possibly a generalised Bayes estimator otherwise. It stems from a hierarchical prior modelling $$\underbrace{\theta|\lambda\sim\mathcal N_p(0,\lambda^{-1}(1-\lambda) I_p)}_\text{first level}\qquad \underbrace{\lambda\sim \pi(\lambda)\propto (c+1)\lambda^{-c}}_\text{second level}$$with the hierarchical Bayes estimator above expressed as $$\delta_c(x)=\left( 1 - \mathbb E[\lambda|x]\right)x$$

A highly interesting if not completely connected paper by Petrone, Rousseau and Scricciolo (2012) studies the connection between empirical Bayes "posteriors" and actual Bayes posteriors, deriving approaches to asymptotically agree (in the number of parameters $n$). They first point out a relevant counterexample of Scott and Berger (2010), for variable selection in regression models.

...consider a Bayesian approach where variable selection is based on a vector of inclusion $γ∈{0,1}^k$ which selects among k potential regressors, and the prior on $γ= (γ_1, . . . , γ_k)$ assumes that the $γ_i$ are independent Bernoulli with parameter $λ$. Scott and Berger (2010) compare this empirical Bayes approach with a hierarchical Bayes procedure that assigns a prior on λ. Surprisingly, they (...) show that the empirical Bayes posterior distribution on the set of models that can be degenerate on the null model ($γ= (0, . . . ,0)$) or on the full model ($γ= (1, . . . ,1)$).

They also produce general conditions for the empirical Bayes posterior to be consistent, which is a necessary condition for (asymptotic) admissibility. And study as an example the Normal mean problem when $$X_i\sim\mathcal N(\theta,1)\qquad\theta|\lambda\sim\mathcal N(\mu,\tau^2)$$ with three situations

1. $\mu=\lambda$ and $\tau$ fixed, in which case the MLE of $\mu$ is $\bar X_n$ and the empirical Bayes posterior is consistent
2. $\tau=\lambda$ and $\mu$ is fixed, in which case the empirical Bayes posterior is only consistent when $\mu$ differs from the true value of the parameter, $m\ne\theta_0$
3. $\lambda=(\mu,\tau)$, in which case the empirical Bayes posterior is degenerate at $\bar X_n$

As noted in this other answer of mine, there may exist admissible estimators that are not Bayes or generalised Bayes estimators.