2
$\begingroup$

I have just worked through an example where, with an improper prior, the bayesian estimator equals the maximum likelihood estimator, leading me to believe that improper priors are uninformative. But must this be true?

$\endgroup$
  • $\begingroup$ Can you define what improper prior is? It is a little ambiguous. $\endgroup$ – Bill Chen Dec 4 '19 at 17:38
  • 4
    $\begingroup$ I've always taken "improper" to mean any prior that isn't a valid probability distribution. Do you mean a uniform prior? $\endgroup$ – shadowtalker Dec 4 '19 at 18:10
  • 1
    $\begingroup$ It is not clear to me that the Bayesian estimator should be the mode of the posterior distribution as I cannot see a realistic loss function which justifies this (except perhaps for saying that the loss for any error does not depend on the size of the error) $\endgroup$ – Henry Dec 4 '19 at 18:46
  • $\begingroup$ I’m talking about a prior distribution that is not a true distribution. $\endgroup$ – David Dec 4 '19 at 20:39
  • 1
    $\begingroup$ My concern was not with the word "improper" which I think is clear enough, but rather with informative; what would convince you that a given prior was informative? $\endgroup$ – Glen_b -Reinstate Monica Dec 5 '19 at 7:50
10
$\begingroup$

...example where, with an improper prior, the Bayesian estimator equals the maximum likelihood estimator...

There is a fundamental issue with this property, namely that it depends on the parameterisation of the sampling model. Indeed, if $$\hat\theta^\text{MAP}=\arg\max_\theta L(\theta|x)\pi(\theta)=\arg\max_\theta L(\theta|x)=\hat\theta^\text{MLE}\tag{1}$$ consider the change of parameterisation $\eta=h(\theta)$, where $h$ is a bijection. Then the prior on $\eta$ is $$\tilde\pi(\eta)=\pi(h^{-1}(\eta))\ \underbrace{\left|\dfrac{\text{d}h^{-1}(\eta)}{\text{d}\eta}\right|}_\text{Jacobian}$$ and there is no reason for $$\hat\eta^\text{MAP}=\arg\max_\eta L(\eta|x)\tilde\pi(\eta)=\arg\max_\eta L(\eta|x)=\hat\eta^\text{MLE}$$ to hold, even though $$\hat\eta^\text{MLE}=h(\hat\theta^\text{MLE})$$ because of the Jacobian. MAP estimators are not invariant by reparameterisation.

As an additional remark, that a particular prior leads to this (dubious) identity is not one of the definitions (here, there, or there) of non-informative priors, since, for one thing, being non-informative is a property that depends on both the data (I assume (1) holds for a particular $x$) and the parameterisation. Furthermore, improper priors should not be assimilated to non-informative priors.

$\endgroup$
8
$\begingroup$

The prior which is uniform over $[m,\infty)$ is improper, but informative: it contains the information that the value is at least $m$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.