3
$\begingroup$

I'm a little confused why everyone writes a flat prior as $f(\theta) \propto c$. In this instance couldn't they just write $f(\theta)=c$? A uniform distribution always a has a constant density function, and AFAICT having a flat prior distribution means having a uniform distribution. Is it because of the case where the set of possible parameter values is infinite? I don't know how a uniform distribution over the entire real line would work or be defined.

$\endgroup$
  • 2
    $\begingroup$ It's just a convenient way of saying that the density doesn't depend on $\theta$ without having to worry about the actual range of values or whether we've specified a "true" density function. It's also consistent with other settings where we drop the the normalizing factor when talking about a conditional distribution in Bayesian statistics. $\endgroup$ – dsaxton Aug 27 '19 at 2:49
12
$\begingroup$

I beg to disagree with the answer given by pche8701: the main reason a flat prior is introduced (in an improper setting) as $f(\theta)\propto c$ or $f(\theta)\propto 1$ which is equivalent but more rigorous is that (i) any constant $c$ leads to the same posterior distribution and (ii) there is no principled way to choose a value for the constant $c$ since a constant density integrates to infinity. It simply cannot be normalised. Hence the qualificative of improper, since it is not a probability density. This explains for instance why improper priors cannot be used in model choice, because the constant $c$ then gets in the way.

While this may appear as a limiting case of a Uniform distribution, exploiting the analogy may lead to paradoxes and contradictions. A flat prior is not a Uniform distribution, but a $\sigma$-finite measure. (Again, I take the convention that one would not use the "flat" denomination in a compact setting since the "uniform" denomination would then become appropriate.)

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ You have a very nice answer but could you clarify the point of disagreement with my answer. I feel we are expressing mostly the same idea. Thanks! $\endgroup$ – tisPrimeTime Aug 27 '19 at 13:40
  • $\begingroup$ I would have imagined that my second paragraph had spelled the difference out rather clearly, namely that there is no Uniform distribution or argument involved in either the flat prior of the unspecified constant $c$. $\endgroup$ – Xi'an Aug 27 '19 at 13:44
  • 2
    $\begingroup$ Sure but my intention wasnt to say a flat prior is necessarily a uniform distribution, but I just provided an example of one to help with understanding. Otherwise I mainly follow along with your fist paragraph in my explanation, unless I misunderstood something? $\endgroup$ – tisPrimeTime Aug 27 '19 at 13:51
  • $\begingroup$ It only integrates to infinity when the bounds for the integral are infinite. So am I right in thinking this is really only necessary in that case? A Bernoulli variable for example only has one parameter that can only be between 0 and 1, there I see no problem with $f(p)=1$. $\endgroup$ – Joseph Garvin Aug 27 '19 at 15:12
  • $\begingroup$ Or are we saying the problem is that $f(p)=1$ as stated doesn't have bounds? $\endgroup$ – Joseph Garvin Aug 27 '19 at 15:13
3
$\begingroup$

As dsaxton said in his comment, it is a convenient way of saying you are working with a flat shaped distribution without having to specify the parameters of the flat prior (e.g. the upper and lower limits of the uniform distribution).

Writing it explicitly as $f(\theta) = c$ would be technically incorrect since the assumption is then that $f(\theta)$ goes on forever in both directions: $(-\infty,\infty)$. You are saying it is literally just the straight line extending to infinity. You cannot have this in probability (well you can, but that is known as an improper prior - a prior distribution which does not integrate to unity, but this is also a bit contentious).

So in general writing it as $f(\theta) \propto c$ removes the ambiguity of having it need to integrate to unity at this point in time - during model specification (since the function is not fully specified yet, it is not equality), and also saves you having to fully parameterise the density in some form, $\mathcal{U}(a,b)$. It leaves things more flexible, and general, without violating any principles of probability.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ One obvious difficulty in this situation is that any value of $c$ violates the probability axioms. Either you have to allow the prior to be (say) a positive measure, but not one that integrates to unity (so it's not a true probability distribution), or you have to understand it as a limiting process related to a sequence of bona fide distributions. $\endgroup$ – whuber Aug 27 '19 at 13:32
  • $\begingroup$ What do you mean by a sequence of bona fide distributions? $\endgroup$ – tisPrimeTime Aug 27 '19 at 13:59
  • 2
    $\begingroup$ Distributions that integrate to unity. An example of such a sequence is the Uniform$(-n,n)$ distribution, $n=1, 2, 3,\ldots.$ $\endgroup$ – whuber Aug 27 '19 at 14:03
2
$\begingroup$

There are various kinds of 'flat' or 'noninformative' prior distributions. A couple of specific examples may help to illustrate both the intuitive idea and technical aspects.

Beta-Binomial. Suppose you have binomial data, such as $x$ successes out of $n$ independent trials, where unknown parameter $\theta$ is the probability of success on any one trial. The binomial likelihood can be written as $f(x|\theta) = {n \choose x}\theta^x(1-\theta)^{n-x}$ and considered as a function of $\theta$ for data $x.$ Often the 'norming constant' ${n \choose x},$ which does not contain $\theta$ is omitted, and one writes $f(x|\theta) \propto \theta^x(1-\theta)^{n-x},$ where the symbol $\propto$ (read "proportional to") indicates the absence of the norming constant.

In this situation one may choose the minimally informative prior distribution $\mathsf{Unif}(0,1) \equiv \mathsf{Beta}(1,1),$ which has density function. $f(\theta) \propto 1 = \theta^0(1-\theta)^0.$ When the norming constant is omitted, this may be called the 'kernel' of the prior distribution.

Multiplying the kernels of the prior and likelihood, one has the posterior kernel $$p(\theta|x) \propto p(\theta) \times p(x|\theta)\\ \propto \theta^{1-1}(1-\theta)^{1-1}\times \theta^x(1-\theta)^{n-x},\\ \propto \theta^{1+x-1}(1-\theta)^{1+n-x -1},$$ where we recognize the last expression as the kernel of the distribution $\mathsf{Beta}(1+x, 1+n-x).$

For technical reasons, some people prefer to use the Jeffreys noninformative prior $\mathsf{Beta}(.5, .5)$ with kernel $f(\theta) \propto \theta^{.5-1}(1-\theta)^{.5-1},$ instead of the uniform prior. Then the posterior posterior distribution turns out to be $\mathsf{Beta}(.5+x. .5+m-x).$ In many cases, the practical difference between the two priors is very small and inconsequential. But with the Jeffreys prior, it would be incorrect to write $f(\theta) \propto 1.$ So not every 'noninformative' prior should be written with $\propto 1.$

Gamma-Poisson. Suppose your data are Poisson counts, such as $x_1, x_2, \dots x_n$ with $t = \sum_{i=1}^n x_i,$ where the unknown parameter $\lambda$ is the Poisson rate. The Poisson likelihood can be written as $f(x|\lambda) \propto \prod_{i=1}^n \lambda^{x_i}e^{-\lambda} \propto \lambda^te^{-n\lambda}.$

An informative prior might be $\mathsf{Gamma}(\text{shape}=4,\text{rate}=1/3),$ which has kernel $f(\lambda) \propto \lambda^{4-1}e^{-(1/3)\lambda}.$ Then the kernel of the posterior distribution is given by $$f(\lambda|x) = f(\lambda)\times f(x|\lambda) \propto \lambda^{4-1}e^{-(1/3)\lambda} \times \lambda^t e^{-n\lambda}\\ \propto \lambda^{[4+t]-1}e^{-[(1/3)+n]\lambda}.$$ where we recognize the last expression as the kernel of $\mathsf{Gamma}(4+t, (1/3)+n).$

If we want a noninformative prior distribution--that has very little effect on the posterior distribution--we can reduce the shape parameter to very near 0 and the rate parameter to very near 0. There is no distribution $``\mathsf{Gamma}(0,0),"$ but we can imagine such an 'improper' gamma distribution. [In a computer application that requires 'legal' parameters, we might use something like 0.0001 for each parameter.] The consequence would be that the likelihood function (essentially) becomes the posterior distribution.

enter image description here

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ It's noninformative and not a distribution. Plugging in 0's for shape and rate parameters does give $1/\lambda.$ $\endgroup$ – BruceET Aug 27 '19 at 17:15
  • $\begingroup$ I'm nor claiming it is 'flat'. I said it is noninformative. $\endgroup$ – BruceET Aug 28 '19 at 7:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.