See this question.
Is this always true? Can the posterior mean always be expressed as a weighted sum of the maximum likelihood estimate and the prior mean (after choosing some appropriate prior)?


1 Answer 1


In (parameter) dimension one, since $$\mathbb E[\theta | x_1,\ldots,x_n]= \dfrac{\mathbb E[\theta | x_1,\ldots,x_n]}{\hat{\theta}(x_1,\ldots,x_n)+\mathbb E[\theta]}\hat{\theta}(x_1,\ldots,x_n)+\dfrac{\mathbb E[\theta | x_1,\ldots,x_n]}{\hat{\theta}(x_1,\ldots,x_n)+\mathbb E[\theta]}\mathbb E[\theta]$$ it is formally always possible.

To quote from an earlier answer of mine to an earlier question,

when $\theta$ is of dimension one, it is always possible to write $$\mathbb E[\theta|\mathbf x] = w(\mathbf x) \mathbb E[\theta] + (1-w(\mathbf x)) \hat\theta(\mathbf x)$$ by solving in $w(x)$ but (i) there is no reason for $0\le w(x)\le 1$ and (ii) this representation does not extend to larger dimensions as $w(\mathbf x)$ will vary for each component.

                                             enter image description here

For exponential families, it is a generic property (see Diaconis and Ylvisaker, 1979, and my undergrad course slide above) that the posterior expectation of the mean of the natural statistic is a convex combination of the prior expectation and the maximum likelihood estimate. (The question you refer to is a special case.) Note however that this does not transfer to the posterior mean of any transform $\phi(\theta)$ of the mean parameter $\nabla\psi(\theta)$ since the expectation of the transform is not the transform of the expectation (another slide of my undergrad course!), while the maximum likelihood estimate of the transform is the transform of the maximum likelihood estimate.

Diaconis and Ylvisaker, 1979 actually show a reciprocal to the above result, namely that if the posterior expectation of $\nabla\psi(\theta)$ is linear in the natural sufficient statistic with fixed weights then the prior is necessarily conjugate:

enter image description here

In a general setting there is no reason for the posterior mean to be located "between" the prior mean and the maximum likelihood. Consider a situation where

  1. the likelihood is multimodal, with the highest mode (i.e., the maximum likelihood estimate $\hat\theta_1$) being very narrow and with another local mode $\hat\theta_2$ being quite widespread
  2. the prior is multimodal, with the prior mean being located on a modal region where the likelihood is essentially zero, and a second modal region $A_2$ covering the second likelihood mode $\hat\theta_2$

the posterior mean could then be located near $\hat\theta_2$, away from both the prior mean and the maximum likelihood estimate $\hat\theta_1$.

  • $\begingroup$ Here, do the coefficients of $\theta_{MLE}$ and $\theta_{prior}$ sum up to $1$? $\endgroup$
    – gunes
    Nov 27, 2020 at 8:23
  • $\begingroup$ @gunes: they do not, indeed, but my point with this remark is to signify that the question is too vague. (Formally, the notion of a weighted sum does not impose the weights to sum up to one.) $\endgroup$
    – Xi'an
    Nov 27, 2020 at 8:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.