3
$\begingroup$

I am curious if there are some known examples of Bayesian models where the prior distribution chosen is itself intractable. That is, suppose we have the data distribution $Y | \theta \sim F(\theta)$ and $\theta \sim \pi(\theta)$ with $$ \pi(\theta) = c \tilde{\pi}(\theta)\, $$ where $\tilde{\pi}(\theta)$ is known, but $c$ is unknown. This will obviously result in the posterior distribution being doubly intractable. But doubly intractable posterior distributions are often so due to the intractability of the likelihood. This makes me curious if there is known work for when we must assume a prior distribution that is intractable.

$\endgroup$
3
  • $\begingroup$ A simple example is a prior resulting from an earlier experiment with an intractable likelihood. But when choosing a prior stricto sensu there is enough flexibility to pick one that is manageable. $\endgroup$
    – Xi'an
    Jun 29, 2020 at 7:25
  • 1
    $\begingroup$ @Xi'an Ah, sure. That example is valid, although a little underwhelming. $\endgroup$ Jun 29, 2020 at 8:10
  • 1
    $\begingroup$ Another situation I can think of is when the parameter space $\Theta$ is defined by a series of complex constraints. For instance when the roots of a associated polynomial $P_\theta(\cdot)$ all are outside the unit circle. Or when the solutions of an attached ODE are all stable. Or when it is the solution of an NP optimisation program. $\endgroup$
    – Xi'an
    Jun 29, 2020 at 12:09

1 Answer 1

2
$\begingroup$

Here is an example that I'm aware of, although surely one can think of others on the same line.

The goal is to encourage separation between the means of a mixture, the model is stated as $$ y_i \sim \sum_{j=1}^H w_j N(\cdot \mid \mu_j, \sigma^2_j) \\ \sigma^2_j \sim g(\cdot) \quad (iid) \\ w_1, \ldots, w_H \sim Dirichlet(\alpha) \\ \mu_1, \ldots, \mu_H \sim \pi(\mu_1, \ldots, \mu_H) $$

and $\pi(\mu_1, \ldots, \mu_H) \propto \prod_{j=1}^H f(\mu_h) \times \prod_{j<h} h(\mu_j , \mu_h)$ where $h(x, y)$ is a monotonically decreasing function of the distance between $x$ and $y$.

A technical report can be found here: https://arxiv.org/abs/1701.04457

$\endgroup$
5
  • $\begingroup$ It is unclear to me that the prior is intractable: you are able to write it down and the normalising constant is irrelevant for the posterior distribution. $\endgroup$
    – Xi'an
    Jun 29, 2020 at 12:01
  • $\begingroup$ For instance, in the pinball sampler paper we wrote with Kerrie Mengersen, we take $f$ as a Gaussian and $$h(\mu_i,\mu_j)=\exp\{-\alpha||\mu_i-\mu_j||^{-2}\}$$ but this does not prevent us from computing the value of $\\pi(\mathbf \mu)$ and from running an MCMC algorithm. $\endgroup$
    – Xi'an
    Jun 29, 2020 at 12:12
  • $\begingroup$ Well, it is intractable by the definition given by @Greenparker: the normalizing constant $c$ is unknown / unfeasible to compute. I think the real problem arise if $f$ has some hyperparameter and you want to update them as well. $\endgroup$
    – mariob6
    Jun 29, 2020 at 12:15
  • $\begingroup$ @user5609462 this example certainly is in line with my question, although Xi'an is right. I didn't quite have this in mind, since here, the intractability of $\mu$s is not a limiting feature in an typical MCMC run. $\endgroup$ Jun 29, 2020 at 12:57
  • 2
    $\begingroup$ Correct. I can think of two cases when this intractability leads to issues: 1) when you have hyperparameters in $f$ and place a prior on those and 2) when you have a prior on $H$ and want to marginalize out the empty clusters (e.g. similar to amstat.tandfonline.com/doi/full/10.1080/01621459.2016.1255636) to avoid reversible jumps $\endgroup$
    – mariob6
    Jun 29, 2020 at 13:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.