Uniform posterior on bounded space vs unbounded space

Question

According to this answer:

There's no problem with a flat posterior on a bounded space, as here. You just have to start out with a prior that's more spread out than a flat one. What you can't have is a flat posterior on an unbounded space, because that's not a proper distribution.

I was wondering if someone can elaborate on (if and) why flat posterior on an unbounded space is not acceptable and how it differs with the bounded space. An example for the latter is a dirichlet distribution $\mathcal{D}irichlet(\alpha_1,\dots,\alpha_n)$ where $\alpha_1 = \alpha_2=\dots=\alpha_n=1$.

Answer 1

It is not possible to have a flat (uniform) probability distribution on an unbounded space, so in particular it's not possible to have a flat posterior distribution.

If you had a uniform probability density on the entire real line, you would need a function $f(x)$ that integrated to 1 (to be a probability density) but was constant. That's not possible: any constant function integrates to 0 or infinity.

Similarly, if you had a uniform distribution on an infinite set of integers, you'd need the probability mass function $p(n)$ to be equal for all $n$ and add to 1. It can't; if $p(n)$ is equal for all $n$ it must add to zero or infinity.

Analogous problems occur for more complicated spaces where it's meaningful to talk about a distribution being 'flat'.

On a bounded finite-dimensional space, it is possible to have a constant function that integrates to 1, and so a probability distribution can be flat. The Dirichlet distribution, for example, is defined on a $n$-dimensional triangle with area $$\mathrm{B}(\boldsymbol{\alpha})=\frac{\prod_{i=1}^{K} \Gamma\left(\alpha_{i}\right)}{\Gamma\left(\sum_{i=1}^{K} \alpha_{i}\right)}$$ so any constant function has finite integral, and a function $$f(\boldsymbol{\alpha})=1/B(\boldsymbol{\alpha})$$ integrates to 1. The probability distribution for New Zealand Lotto is over the set of six-number sequences with values from 1 to 40, so there are only finitely many of them, and you can put equal probability on each one ($p(x)=1/3838380$) and have it add up to 1.

So, given that, the real question is how flat prior distributions make sense. It turns out that you can often put a constant function into Bayes' Rule in place of the prior density and get a genuine distribution out as the posterior. It makes sense, then, to think of that posterior as belonging to a 'flat prior' even if there is no such thing. Also, the posterior you get for a 'flat prior', when there is one, is often the same as the limit of the posteriors you'd get for more and more spread out genuine priors [I don't know if this is always true or just often true]. So, for example, if you have $X_m\sim N(\mu,1)$ data and a $\mu\sim N(0,\omega^2)$ prior, the posterior is Normal with mean $$\frac{n\bar X_n}{n+\omega^{-2}}$$ and variance $1/(n+\omega^{-2})$. If you let $\omega$ increase, the prior gets more and more spread out and the posterior gets closer and closer to $N(\bar X, 1/n)$, which is also what you'd get with a 'flat prior'.

Sometimes, though, using a 'flat prior' doesn't give a genuine probability distribution for the posterior, in which case it doesn't really make sense.

Answer 2

Strictly speaking, the question is imprecise in that it does not specify the reference measure. If the reference measure is $\text{d}\mu(x)=e^{-x^2}\text{d}\lambda(x)$ where $\lambda$ is the Lebesgue measure, a posterior with a flat density is valid.

Assuming however using a "flat prior" means having a constant density with respect to the Lebesgue measure, Thomas Lumley's answer clearly explains why Bayesian inference is impossible with such a "posterior". This is not a probability density and hence the posterior is simply not defined. There is no way to compute posterior expectations or even posterior probabilities since the posterior mass of the entire space in infinity. Any parameter space with an infinite volume cannot be inferred under a posterior like this. More generally any posterior integrating to infinity is not acceptable for Bayesian inference for the very same reason that this cannot be turned into a probability density.

As a marginalia, and as discussed in an earlier X validated entry, the maximum entropy prior $$\arg_p \max \int p(x) \log p(x) \text{d}\lambda(x)$$ is defined in terms of a dominating measure $\text{d}\lambda$. There is no absolute or unique measure of entropy in continuous spaces.