Does a Bayes estimator require that the true parameter is a possible variate of the prior?

Question

This might be a bit of a philosophical question, but here we go: In decision theory, the risk of a Bayes estimator $\hat\theta(x)$ for $\theta\in\Theta$ is defined with respect to a prior distribution $\pi$ on $\Theta$.

Now, on the one hand, for the true $\theta$ to have generated the data (i.e. "exist"), $\theta$ must be a possible variate under $\pi$, e.g. have non-zero probability, non-zero density, etc.; on the other hand, $\theta$ is not known, hence the choice of a prior, so we have no guarantee that the true $\theta$ is a possible variate under the $\pi$ we chose.

Now, it appears to me that we somehow have to select $\pi$ such that $\theta$ would be a possible variate. Otherwise, certain theorems would not hold. For instance, the minimax estimate would not be a Bayes estimate for a least favorable prior, since we could make that prior arbitrarily bad by excluding a large region around and including $\theta$ from its domain. However, guaranteeing that $\theta$ is indeed in the domain might be hard to achieve.

So my questions are:

1. Is it generally assumed that the actual $\theta$ is a possible variate of $\pi$?
2. Can this be guaranteed?
3. Can cases violating this at least be detected somehow, so one does not rely on theorems such as minimax when the conditions don't hold?
4. If it is not required, why do the standard results in decision theory hold then?

Answer 1

Very nice question! It would indeed make sense that a "good" prior distribution gives positive probability or positive density value to the "true" parameter $\theta_0$, but from a purely decisional perspective this does not have to be the case. A simple counter-example to this "intuition" that$$\pi(\theta_0)>0$$should be necessary, when $\pi(\cdot)$ is the prior density and $\theta_0$ is the "true" value of the parameter, is the brilliant minimaxity result of Casella and Strawderman (1981): when estimating a Normal mean $\mu$ based on a single observation $x\sim{\cal N}(\mu,1)$ with the additional constraint that $|\mu|<\rho$, if $\rho$ is small enough, $\rho\le 1.0567$ specifically, the minimax estimator corresponds to a (least favourable) uniform prior on $\{-\rho,\rho\}$, meaning that $\pi$ gives equal weight to $-\rho$ and $\rho$ (and none to any other value of the mean $\mu$) $$\pi(\theta)=\frac{1}{2}\delta_{-\rho}(\theta)+ \frac{1}{2}\delta_{\rho}(\theta)$$ When $\rho$ increases the least favourable prior sees its support growing, but remaining a finite set of possible values. However the posterior expectation, $\mathbb{E}[\mu|x]$, can take any value on $(-\rho,\rho)$.

The core of the discussion (see comments) may be that, were the Bayes estimator to be constrained to be a point in the support of $\pi(\cdot)$, its properties would be quite different.

Similarly, when considering admissible estimators, Bayes estimators associated with a proper prior on a compact set are usually admissible, although they have a restricted support.

In both cases, the frequentist notion (minimaxity or admissibility) is defined over the possible range of parameters rather that at the "true" value of the parameter (which brings an answer to Question 4.) For instance, looking at the posterior risk $$\int_\Theta L(\theta,\delta) \pi(\theta|x)\text{d}\theta$$ or at the Bayes risk $$\int_{\cal X}\int_\Theta L(\theta,\delta) \pi(\theta)f(x|\theta)\text{d}\theta\text{d}x$$ does not involve the true value $\theta_0$.

Furthermore, as pointed out in the above example, when the Bayes estimator is defined by a formal expression such as the posterior mean $$\hat{\theta}^\pi(x)=\int_\Theta \theta\pi(\theta|x)\text{d}\theta$$ for the quadratic (or $L_2$) loss, this estimator may take values outside the support of $\pi$ in cases this support is not convex.

As an aside, when reading

for the true θ to have generated the data (i.e. "exist"), θ must be a possible variate under π, e.g. have non-zero probability, non-zero density

I consider it a misrepresentation of the meaning of a prior. The prior distribution is not supposed to stand for an actual physical (or real) mechanism that saw a parameter value $\theta_0$ generated from $\pi$ followed by an observation $x$ generated from $f(x|\theta_0)$. The prior is a reference measure on the parameter space that incorporates prior information and subjective beliefs about the parameter and that is by no means unique. A Bayesian analysis is always relative to the prior chosen to conduct this Bayesian analysis. Hence, there is not an absolute necessity for the true parameter to belong to the support of $\pi$. Obviously, when this support is a compact connected set, ${\mathscr A}$, any value of the parameter outside the set ${\mathscr A}$ cannot be consistently estimated by the posterior mean $\hat{\theta}^\pi$ but this does not even prevent the estimator to be admissible.

Answer 2

1. Yes, it is generally assumed that the true $\theta$ is in the domain of the prior. It is the responsibility of the statistician to see that this is the case.

2. Usually, yes. For example, when estimating a mean or location parameter, any prior on $(-\infty, \infty)$ will have the true value in its domain. (If the parameter is known to be greater than zero, e.g., "mean number of traffic accidents on the Bay Bridge per day", the prior doesn't need to include negative values, obviously.) If we are estimating a probability, any prior on $[0,1]$ will have the true value in its domain. If we are constructing a prior on a variance term, any prior on $(0, \infty)$ will have the true value in its domain... and so on.

3. If your posterior is "stacked up" at one edge of the domain of the prior, and your prior imposes an unnecessary restriction on the domain at that same edge, this is an ad-hoc indicator that the unnecessary restriction may be causing you problems. But this should only occur if a) you have constructed a prior whose form is driven largely by convenience instead of actual prior knowledge, and b) the convenience-induced form of the prior restricts the domain of the parameter to a subset of what its "natural" domain can be considered to be.

An example of such is an old, hopefully long obsoleted, practice of bounding the prior on a variance term slightly away from zero in order to avoid potential computational difficulties. If the true value of the variance is between the bound and zero, well... but actually thinking about the potential values of the variance given the data, or (for example) putting the prior on the log of the variance instead, will allow you to avoid this problem, and similar mild cleverness should allow you to avoid domain-limiting priors in general.

4. Answered by #1.

Answer 3

The simple, intuitive answer is that prior reflects your prior knowledge about the $\theta$ and the minimal knowledge that you should have, is about it's domain. If you use bounded prior, then you assume that the values outside the bounds have zero probability, are impossible, and this is a very strong assumption that should not be made without good rationale. This is why people who don't want to make strong prior assumptions, use vague priors on $-\infty$ to $\infty$.

Besides the bounded case, when your sample grows, or more precisely conveys more information, your posterior should finally converge to $\theta$ no matter of prior.