# Prior distributions letting a small sample “speak”

I’ve got a general question.

Let k be a parameter which must be estimated. It lies within the interval $[a, b]$, $a$ and $b$ being finite real numbers.

Let us further assume we dispose of a series of measurements x of known standard deviations. $X$ is a complex function of $k$.

Do we dispose of some type of guarantee that Jefrreys prior imitates the shape of the likelihood in such a way that the posterior distribution delivers us a result close enough to the Maximum Likelihood Estimate?

If not, what about Bernardo’s prior?

I’m asking this question for cases where only few measurements are at hand.

In such a situation, I personally would simply use the likelihood $l_{x(k)}$ in order to define the prior as $f_0(k) = l_x(k)/I$, I being the integral of $l_x$ over the possible values of $k$ (which is a closed interval).

While this doubtlessly sounds very scandalous to any orthodox Bayesian, this has the great advantage of preventing an unphysical prior from erasing the influence of the experimental data on the posterior.

I’d be very interested to know, however, if either Jeffreys prior or Bernardo’s prior could do the job as well.

• The prior has nothing to do with the likelihood. What does it mean to define the prior as proportional to the likelihood? – Neil G Aug 31 '15 at 8:02

You can read all about what Jaynes said about Prior Probabilities here: Jaynes, E. T. (1968). Prior probabilities. IEEE Transactions on Systems Science and Cybernetics, 4(3), 227–241. You should be able to find a free copy on the web. (The conclusion is probably the best part.)

I don't know what you mean by setting the prior to be proportional to the likelihood. The prior is something you have before you have a likelihood.

It's very easy for the maximum likelihood estimate to be different than maximum a posteriori estimate if your prior and your likelihood are different.

I disagree with the intuition that the maximum likelihood estimate is somehow ideal.

I am much more convinced by Jaynes' argument that the prior should be invariant under reparametrization since the parametrization is arbitrary. The uniform prior is not invariant. (Note that the maximum likelihood estimate is invariant under reparametrization.) Therefore, I like Jeffreys prior best.

• Thank you very much for your answer Neil. I'm well aware of the work of Jayne. The prior represents a unique belief distribution BEFORE the arrival of the data only if you are a traditional objective Bayesian. If you are a heretical Bayesian, a subjective Bayesian or a proponent of imprecise probabilities, you may very well view these priors as useful conventions which should be chosen in such a way that the posterior is similar to the likelihood. – Marc Lüttingen Aug 31 '15 at 12:41
• I understand this doesn't make any sense from your own standpoint. My main question is: has Jeffreys prior distribution most often the same shape as the likelihood? Or can it be very different from it under some circumstances? I guess that the latter is the case but I'm not entirely sure. – Marc Lüttingen Aug 31 '15 at 12:42
• @MarcLüttingen Like you say, this doesn't make any sense to me. The Jeffreys prior is chosen to be as minimally assumptive as possible. Sure the Jeffreys prior, or any prior that is chosen before the data can look nothing like the data. If the data can't change your belief, why even bother with it? Just return your prior. If what you're calling the prior involves the data, then what is your belief before that? My suggestion is to use the terminology "prior" for what comes before that — and "likelihood" for the data — and then we'll be on the same page. – Neil G Aug 31 '15 at 12:48