Prior distributions letting a small sample "speak"

Question

I’ve got a general question.

Let k be a parameter which must be estimated. It lies within the interval $[a, b]$, $a$ and $b$ being finite real numbers.

Let us further assume we dispose of a series of measurements x of known standard deviations. $X$ is a complex function of $k$.

Do we dispose of some type of guarantee that Jefrreys prior imitates the shape of the likelihood in such a way that the posterior distribution delivers us a result close enough to the Maximum Likelihood Estimate?

If not, what about Bernardo’s prior?

I’m asking this question for cases where only few measurements are at hand.

In such a situation, I personally would simply use the likelihood $l_{x(k)}$ in order to define the prior as $f_0(k) = l_x(k)/I$, I being the integral of $l_x$ over the possible values of $k$ (which is a closed interval).

While this doubtlessly sounds very scandalous to any orthodox Bayesian, this has the great advantage of preventing an unphysical prior from erasing the influence of the experimental data on the posterior.

I’d be very interested to know, however, if either Jeffreys prior or Bernardo’s prior could do the job as well.

Provided it doesn’t cost you too much time, I’d be delighted to read your answer.

Answer 1

You can read all about what Jaynes said about Prior Probabilities here: Jaynes, E. T. (1968). Prior probabilities. IEEE Transactions on Systems Science and Cybernetics, 4(3), 227–241. You should be able to find a free copy on the web. (The conclusion is probably the best part.)

I don't know what you mean by setting the prior to be proportional to the likelihood. The prior is something you have before you have a likelihood.

It's very easy for the maximum likelihood estimate to be different than maximum a posteriori estimate if your prior and your likelihood are different.

I disagree with the intuition that the maximum likelihood estimate is somehow ideal.

I am much more convinced by Jaynes' argument that the prior should be invariant under reparametrization since the parametrization is arbitrary. The uniform prior is not invariant. (Note that the maximum likelihood estimate is invariant under reparametrization.) Therefore, I like Jeffreys prior best.