Assume that $\theta$ is an unknown parameter with prior belief $p(\theta)$ and that $X_1,...,X_n$ is an iid sample from $p(x|\theta)$.

What does it say about our prior distribution $p(\theta)$ when the Bayesian estimator $E[\theta|X=x]$ coincides with the maximum likelihood estimator $\hat{\theta}$? My professor posed this question in his notes and I don't know the answer.

Edit: The question was posed following an example where the prior distribution was improper, but not uniform.

  • $\begingroup$ Have a look for, e.g., uniform priors. $\endgroup$ – Christoph Hanck Jan 15 '20 at 14:08
  • $\begingroup$ The question was based on an example where we had an improper prior distribution, but it was not a uniform improper prior. I have now included this info in the question. $\endgroup$ – Dion Jan 15 '20 at 14:42
  • $\begingroup$ The prior is flat in the region of the MLE and doesn't have sufficiently large modes in other regions to overcome the relatively smaller values of the likelihood function in those regions, mainly. $\endgroup$ – jbowman Jan 15 '20 at 16:14
  • 2
    $\begingroup$ The question asks about the implications of posterior expectation (not mode of posterior) and mode of the likelihood being equal. I think we need to reopen it. $\endgroup$ – gunes Jan 15 '20 at 16:30
  • $\begingroup$ @haventureinstatedmonicayet The answer is the same though: MLE is invariant by reparameterisation, posterior mean is not. $\endgroup$ – Xi'an Jan 15 '20 at 19:04

Consider a binomial model with beta prior $\theta\sim Beta(\alpha,\beta)$ for which we have observed $k$ successes in $n$ attempts. The MLE is $\hat\theta=k/n$, and the posterior mean is $$ E(\theta|y)=\frac{\alpha+k}{\alpha+\beta+n} $$ We see that $E(\theta|x)=\hat\theta$ if $\alpha=\beta=0$. This is known as the Haldane prior $\pi(\theta)\propto \theta^{-1}(1-\theta)^{-1}$, which is improper in view of $\int\theta^{-1}(1-\theta)^{-1}d\theta=\ln(\theta)-\ln(1-\theta)+C$ (and strictly speaking not a Beta prior, as it does not have positive parameters). Plotting it (in fact, an approximation with $\alpha,\beta\approx0$) shows it to have an extreme shape of a bathtub, putting all prior density on either 0 or 1: enter image description here

theta <- seq(0,1,by=.000001)
plot(theta, dbeta(theta,.001,.001), type="l", lwd=2,
     col="lightblue", ylim=c(0,10), ylab="Haldane approximation")

That leads to a behavior in which, when $k=n$ ($k=0$), we only expect successes (failures) in the future, as the posterior then "switches" to the value of $\theta$ that receives support by the data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.