In my problem, I have two sets of parameters, $\theta_1$ and $\theta_2$, and two datasets $d_1,d_2$ that constrain them with a known likelihood function. There is a certain 'hierarchy' in the model: the likelihood is separable and one only depends on $\theta_1$: $$\mathcal{L}(d_1,d_2|\theta_1,\theta_2) = \mathcal{L}(d_2|\theta_1,\theta_2)\mathcal{L}(d_1|\theta_1).$$ I want to sample from the joint posterior $$p(\theta_1, \theta_2 | d_1, d_2).$$ Unfortunately, due to the high cost of evaluating $\mathcal{L}(d_1|\theta_1)$, it is not practical to directly sample this posterior. I am lucky: the likelihood of $d_1$ is only a function of $\theta_1$. Because of that, I want to split up this posterior in a hierarchical way like so: $$p(\theta_1, \theta_2 | d_1, d_2) \propto \mathcal{L}(d_2,d_1|\theta_1,\theta_2)\pi(\theta_1,\theta_2)\\ =\mathcal{L}(d_2|\theta_1,\theta_2) (\mathcal{L}(d_1|\theta_1)\pi(\theta_1))\pi(\theta_2).$$

This suggests doing the sampling in two steps: first sampling from $p(\theta_1|d_1)=\mathcal{L}(d_1|\theta_1)\pi(\theta_1)$ and subsequently using this as a prior on the full posterior. Computationally, this would be much preferred.

However, in practice, my $p(\theta_1|d_1)$ is not analytically calculable, so instead it is sampled, with e.g. MCMC. But if you only have samples, how do you use this as a prior in an MCMC run of $p(\theta_1,\theta_2|d_1,d_2)$? You have to parameterize this prior somehow, right?

A simple way would be to assume that $p(\theta_1|d_1)$ is a multivariate Gaussian, with a mean and covariance matrix derived from the sampled MCMC chains, but this is not always a reasonable assumption. If parametrizing is indeed necessary, how do you do this if the posterior is non-gaussian, for example a banana-shape, in some parameters?

  • $\begingroup$ I think in a typical hierarchical model the priors for $\theta_i$ would involve hyperparameters such as $\psi$. The joint prior for all the unknowns would have the form $p(\psi)\,\prod_i p(\theta_i|\psi)$. This prior induces dependence between $\theta_i$ and $\theta_j$. It is via this dependence that the information $d_i$ informs $\theta_j$ for example. Is this the sort of model you have in mind? $\endgroup$ – mef Nov 22 '20 at 12:55
  • $\begingroup$ Not exactly, in my case the priors $\pi(\theta_i)$ are already fully known and independent of parameters. Perhaps, this is not a strict hierarchical model then, sorry... I'll edit the question a bit! $\endgroup$ – Ewoud Nov 22 '20 at 17:37
  • 1
    $\begingroup$ Okay, I see. I have a question: If evaluating $\mathcal{L}(d_1|\theta_1)$ is so expensive, how can you sample from $p(\theta_1|d_1)$? $\endgroup$ – mef Nov 22 '20 at 18:36
  • $\begingroup$ Two reasons, firstly because $\theta_1$ has only a dimension of 4, but $\theta_2$ has a dimension of 12. And secondly, because I could reuse this $p(\theta_1|d_1)$: as a next step I want to compare the posteriors for a set of, say 100 different $d_2$'s. $\endgroup$ – Ewoud Nov 22 '20 at 19:09
  • $\begingroup$ I'm sorry, I don't see how that answers my question. I must be missing something. Good luck. $\endgroup$ – mef Nov 22 '20 at 19:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.