5
$\begingroup$

According to this answer:

There's no problem with a flat posterior on a bounded space, as here. You just have to start out with a prior that's more spread out than a flat one. What you can't have is a flat posterior on an unbounded space, because that's not a proper distribution.

I was wondering if someone can elaborate on (if and) why flat posterior on an unbounded space is not acceptable and how it differs with the bounded space. An example for the latter is a dirichlet distribution $\mathcal{D}irichlet(\alpha_1,\dots,\alpha_n)$ where $\alpha_1 = \alpha_2=\dots=\alpha_n=1$.

$\endgroup$
0

2 Answers 2

13
$\begingroup$

It is not possible to have a flat (uniform) probability distribution on an unbounded space, so in particular it's not possible to have a flat posterior distribution.

If you had a uniform probability density on the entire real line, you would need a function $f(x)$ that integrated to 1 (to be a probability density) but was constant. That's not possible: any constant function integrates to 0 or infinity.

Similarly, if you had a uniform distribution on an infinite set of integers, you'd need the probability mass function $p(n)$ to be equal for all $n$ and add to 1. It can't; if $p(n)$ is equal for all $n$ it must add to zero or infinity.

Analogous problems occur for more complicated spaces where it's meaningful to talk about a distribution being 'flat'.

On a bounded finite-dimensional space, it is possible to have a constant function that integrates to 1, and so a probability distribution can be flat. The Dirichlet distribution, for example, is defined on a $n$-dimensional triangle with area $$\mathrm{B}(\boldsymbol{\alpha})=\frac{\prod_{i=1}^{K} \Gamma\left(\alpha_{i}\right)}{\Gamma\left(\sum_{i=1}^{K} \alpha_{i}\right)}$$ so any constant function has finite integral, and a function $$f(\boldsymbol{\alpha})=1/B(\boldsymbol{\alpha})$$ integrates to 1. The probability distribution for New Zealand Lotto is over the set of six-number sequences with values from 1 to 40, so there are only finitely many of them, and you can put equal probability on each one ($p(x)=1/3838380$) and have it add up to 1.

So, given that, the real question is how flat prior distributions make sense. It turns out that you can often put a constant function into Bayes' Rule in place of the prior density and get a genuine distribution out as the posterior. It makes sense, then, to think of that posterior as belonging to a 'flat prior' even if there is no such thing. Also, the posterior you get for a 'flat prior', when there is one, is often the same as the limit of the posteriors you'd get for more and more spread out genuine priors [I don't know if this is always true or just often true]. So, for example, if you have $X_m\sim N(\mu,1)$ data and a $\mu\sim N(0,\omega^2)$ prior, the posterior is Normal with mean $$\frac{n\bar X_n}{n+\omega^{-2}}$$ and variance $1/(n+\omega^{-2})$. If you let $\omega$ increase, the prior gets more and more spread out and the posterior gets closer and closer to $N(\bar X, 1/n)$, which is also what you'd get with a 'flat prior'.

Sometimes, though, using a 'flat prior' doesn't give a genuine probability distribution for the posterior, in which case it doesn't really make sense.

$\endgroup$
5
  • $\begingroup$ Indeed! As a related note, the distribution that maximizes entropy for a given variance is the uniform in a discrete space, but the normal on a continuous one $\endgroup$
    – fr_andres
    Aug 18, 2020 at 8:27
  • $\begingroup$ @fr_andres: This result of optimality for the Normal only holds when the maximum entropy is defined in terms of the Lebesgue measure. Using another dominating measure produces another distribution. $\endgroup$
    – Xi'an
    Aug 18, 2020 at 16:27
  • 1
    $\begingroup$ Dear @Xi'an I also read your answer and learned a lot form it, thanks. Got me to learn some fundamental stuff and is also really interesting to know that it affects entropy too. Do you have any interesting examples for that? E.g. if d1 has more entropy than d2 on Lebesgue, can the opposite be in a different measure? Thanks again $\endgroup$
    – fr_andres
    Aug 18, 2020 at 18:01
  • 1
    $\begingroup$ @fr_andres: there is no absolute measure of entropy on $\mathbb R$, see the new links at the bottom of my answer. $\endgroup$
    – Xi'an
    Aug 19, 2020 at 8:21
  • 1
    $\begingroup$ @Xi'an Thanks again for the very educational reaction $\endgroup$
    – fr_andres
    Aug 19, 2020 at 15:18
8
$\begingroup$

Strictly speaking, the question is imprecise in that it does not specify the reference measure. If the reference measure is $\text{d}\mu(x)=e^{-x^2}\text{d}\lambda(x)$ where $\lambda$ is the Lebesgue measure, a posterior with a flat density is valid.

Assuming however using a "flat prior" means having a constant density with respect to the Lebesgue measure, Thomas Lumley's answer clearly explains why Bayesian inference is impossible with such a "posterior". This is not a probability density and hence the posterior is simply not defined. There is no way to compute posterior expectations or even posterior probabilities since the posterior mass of the entire space in infinity. Any parameter space with an infinite volume cannot be inferred under a posterior like this. More generally any posterior integrating to infinity is not acceptable for Bayesian inference for the very same reason that this cannot be turned into a probability density.

As a marginalia, and as discussed in an earlier X validated entry, the maximum entropy prior $$\arg_p \max \int p(x) \log p(x) \text{d}\lambda(x)$$ is defined in terms of a dominating measure $\text{d}\lambda$. There is no absolute or unique measure of entropy in continuous spaces.

$\endgroup$
1
  • 3
    $\begingroup$ Very true. I thought about bringing up translation-invariant measures and decided not to go there. $\endgroup$ Aug 16, 2020 at 6:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.