Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Assume that $\theta$ is an unknown parameter with prior belief $p(\theta)$ and that $X_1,...,X_n$ is an iid sample from $p(x|\theta)$.
What does it say about our prior distribution $p(\theta)$ when the Bayesian estimator $E[\theta|X=x]$ coincides with the maximum likelihood estimator $\hat{\theta}$? My professor posed this question in his notes and I don't know the answer.
Edit: The question was posed following an example where the prior distribution was improper, but not uniform.
Consider a binomial model with beta prior $\theta\sim Beta(\alpha,\beta)$ for which we have observed $k$ successes in $n$ attempts. The MLE is $\hat\theta=k/n$, and the posterior mean is
$$
E(\theta|y)=\frac{\alpha+k}{\alpha+\beta+n}
$$
We see that $E(\theta|x)=\hat\theta$ if $\alpha=\beta=0$. This is known as the Haldane prior $\pi(\theta)\propto \theta^{-1}(1-\theta)^{-1}$, which is improper in view of $\int\theta^{-1}(1-\theta)^{-1}d\theta=\ln(\theta)-\ln(1-\theta)+C$ (and strictly speaking not a Beta prior, as it does not have positive parameters). Plotting it (in fact, an approximation with $\alpha,\beta\approx0$) shows it to have an extreme shape of a bathtub, putting all prior density on either 0 or 1:
theta <- seq(0,1,by=.000001)
plot(theta, dbeta(theta,.001,.001), type="l", lwd=2,
col="lightblue", ylim=c(0,10), ylab="Haldane approximation")
That leads to a behavior in which, when $k=n$ ($k=0$), we only expect successes (failures) in the future, as the posterior then "switches" to the value of $\theta$ that receives support by the data.
