How does Prior Variance Affect Discrepancy between MLE and Posterior Expectation

Question

Suppose that $\theta\in R$ is a parameter of interest, $p(\theta)$ is our prior belief regarding $\theta$, and $\hat \theta$ is the MLE for theta derived from the data $x$. It is my understanding that because the posterior expectation always lies in between the prior expectation $E[\theta]$ and the MLE $\hat \theta$, it can be expressed as a weighted average of the two values. Furthermore, it was my understanding that if $w_{prior}=w$ represents the weight given to $E[\theta]$ and $(1-w)$ represents the weight given to $\hat \theta$, i.e. $E_{\theta|x}[\theta|x]=wE_{\theta}[\theta]+(1-w)\hat \theta$, then $w$ is inversely related to the variance of $p(\theta)$.

Assuming that my understanding is correct (obviously correct me if not), then as the variance of $p(\theta)$ approaches its upper bound, then $w$ approaches zero. The reason I doubt this statement is that there are prior distributions whose variances are bounded above (i.e. Beta distribution), and I don't know if the corresponding weight may be bounded below by a number larger than zero.

Edit: According to @Xi-an's comment, it is not true that the posterior expectation must lie between the prior expectation and the MLE. Could someone provide an example of when this occurs?

Answer 1

As indicated in the previous answer, this linearity with a fixed weight holds when the model is Gaussian with unknown mean and the prior is the conjugate Gaussian model. This is essentially the representative case as

1. the fact that the posterior expectation only depends of the data through the MLE is a form of sufficiency of the MLE that does not stand outside exponential families with conjugate priors. When considering an exponential family in its natural representation $$f(x|\theta)=\exp\{\theta\cdot x - \psi(\theta)\}$$ with a conjugate prior $$\pi(\theta)\propto \exp\{\theta\cdot \mu - \lambda\psi(\theta)\}$$ linearity holds for the mean parameter $\nabla\psi(\theta)$ $$\mathbb E[\nabla\psi(\theta)|x] = \overbrace{w}^{=\lambda/1+\lambda} \underbrace{\mathbb E[\nabla\psi(\theta)]}_{=\mu/\lambda} + (1-w) \underbrace{\widehat{\nabla\psi(\theta)}}_\text{MLE $x$}$$

As shown by Diaconis and Ylvisaker (1979), this is a (less common) characterisation of conjugate measures.

2. the linear relation does not keep under a change of parameterisation: if $$\mathbb E[\theta|x] = w \mathbb E[\theta] + (1-w) \hat\theta$$ for a non-linear one-to-one transform $\varphi$, $$\mathbb E[\varphi(\theta)|x] \ne w \mathbb E[\varphi(\theta)] + (1-w) \varphi(\hat\theta)$$in the vast majority of cases

3. when $\theta$ is of dimension one, it is always possible to write $$\mathbb E[\theta|x] = w(x) \mathbb E[\theta] + (1-w(x)) \hat\theta$$ by solving in $w(x)$ but (i) there is no reason for $0\le w(x)\le 1$ and (ii) this representation does not extend to larger dimensions as $w(x)$ will vary for each component.

Answer 2

I think you have a minor confusion on Bayesian and Frequentist paradigms.

The particular case you are referring to is inference over $\mu$ for $x_i \sim N( \mu , \sigma^2 )$ with $\sigma^2$ is known. In this case, which belongs to an example of conjugate families , the posterior mean ($\mu_p$) from the posterior distribution becomes a convex combination between the prior mean and $\overline{x}$ (The MLE estimator) as a function of the prior variance $\sigma_{\mu}^2$ and the known variance $\sigma^2$.

This estimator has some cool asymptotic qualities, like when $n \longrightarrow \infty$ (all else the same) then $\mu_p \longrightarrow \overline{x}$.

To see the actual computations I recommend this paper.

However this is not always the case for conjugate families, as posterior parameters do not behave as in the gaussian case. Additionally when using MCMC techniques we do not have a closed analytic form for the posterior parameters.