2
$\begingroup$

In Bayesian statistics, how do software packages compute the posterior distribution when the prior is improper (flat)?

If I understand correctly, this can't be done analytically so how is it done computationally?

$\endgroup$
  • 2
    $\begingroup$ This can be done analytically for a number of models, so I don't understand the question. $\endgroup$ – jaradniemi Nov 2 '17 at 21:12
  • 1
    $\begingroup$ Improper priors aren't a problem on their own -- they're a problem when the posterior is improper. (Proper priors aren't necessary for proper posteriors.) $\endgroup$ – Sycorax Nov 2 '17 at 21:13
  • 2
    $\begingroup$ Contemporary applied Bayesian statistics use Markov Chain Monte Carlo [MCMC] and related methods to estimate the posterior distribution. (I would include a link here, but I can't find a good explanation of MCMC quickly.) MCMC requires that the RHS of Bayes' theorem (likelihood x "prior") be easy to calculate at any given point in the parameter space; it doesn't require that this RHS be a probability density. $\endgroup$ – Dan Hicks Nov 2 '17 at 21:16
  • 1
    $\begingroup$ @Sycorax Fixed. I didn't want to touch it without being sure. $\endgroup$ – Glen_b Nov 3 '17 at 1:45
3
$\begingroup$

As pointed out by the comments and by Glen's answer, the OP is mistaken in the impression that improper priors cannot be handled analytically. Any perusal of a Bayesian textbook [like mine!] would dispel this impression.

For instance, take a Normal mean problem with $x\sim\mathcal{N}(\theta,1)$ and the unknown mean $\theta$ endowed with the flat prior $\pi(\theta)=c$ for an arbitrary constant $c$. The posterior distribution is easily derived as $\theta|x\sim\mathcal{N}(x,1)$.

The only relevant point in the question is whether or not software handle improper priors differently and in particular spotting whether or not the posterior does not exist. The answer is that most of the methods and software do not. Monte Carlo methods are equipped for dealing with un-normalised densities and miss the possible occurrence of an infinite integral. MCMC methods similarly use the un-normalised densities as entries and process them in acceptance probabilities with the same danger. This even includes the Gibbs sampler, which may rely on well-defined conditional distributions that have no joint probability distribution. A very simple illustration comes from Casella's & George's Gibbs for Kids: \begin{align*} \eta|\theta&\sim\mathcal{E}(\theta)\\ \theta|\eta&\sim\mathcal{E}(\eta)\\ \end{align*} has no proper stationary measure. One of the first papers by Alan Gelfand and Adrian Smith actually contains a Gibbs sampler for a random effect model with an improper Jeffreys prior that suffers from this major drawback.

This is presumably one reason why the early BUGS (Bayes using Gibbs sampling) software has a prohibition of improper priors. (While advocating replacing these improper priors by proper priors with huge variances is questionable!) Stan certainly allows for improper priors.

$\endgroup$
2
$\begingroup$

As jaradniemi points out, the question relies on a mistaken premise (that there's some analytical difficulty in dealing with improper priors); typically this isn't the case.

As Sycorax says, it's the posterior that's the issue. If the posterior has a finite integral, there should be no particular difficulty (though how a particular posterior is going to be calculated depends on what methodology you're using).

An improper prior doesn't necessarily lead to an improper posterior. While it's possible in some situations to end up with an improper posterior, in many cases of typical practice this is simply not an issue.

As an example we could look at a univariate case -- a flat prior on the real line is improper, but the resulting posterior will simply be a normalized likelihood function, and with typical sorts of problems and nontrivial sample sizes that will generally yield a proper distribution.

In many cases improper priors are no harder to deal with -- and are typically dealt with in the same way -- as a proper prior. For example, generally one doesn't need to sample a prior. If you look at (for one example), the test for whether one accepts a Metropolis-Hastings proposal to move to a new value, or stays put, the acceptance probability is ${ A(x'|x)=\min \left(1,{\frac {f(x')}{f(x)}}{\frac {q(x|x')}{q(x'|x)}}\right)}$ where $q$ is the proposal density and $f$ is proportional to the density you want to sample from (which is typically some joint posterior, but may be a conditional posterior in some situations (such as in a Metropolis-within-Gibbs step).

Note for example that it's only the ratio of $f$ at $x$ and $x'$ that comes into the calculations -- indeed a flat prior would simply cancel out -- but other priors would simply result in typically fairly ordinary scaling ratios (e.g. consider an improper joint prior for a location $\theta$ and scale parameter $\tau$ being $1/\tau$ --the ratio of $f$'s would be a ratio of likelihoods times $\tau/\tau'$, which presents no difficulty at all).

[I'd suggest trying some basic Bayesian calculations - with very simple, tractable problems - using conjugate but improper priors and seeing how that works; this would give some intuition for how one can have a proper posterior]

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.