2
$\begingroup$

The problem

  • Suppose I have $K$ independent probability densities $f_1(\theta), f_2(\theta), \ldots, f_K(\theta)$ defined over a variable $\theta$.
  • I do not have a closed expression for the $f_k$, but they can be computed numerically up to a normalization constant (i.e., I can estimate them via MCMC sampling).
  • My goal is to compute $g_I(\theta) = \frac{1}{Z_I} \prod_{k \in I} f_k(\theta)$, where $I \subseteq \{1,\ldots,K\}$ and $Z_I$ is a normalization constant. I am interested in computing $g_I(\theta)$ for several subsets $I^{(1)},\ldots,I^{(M)}$.
  • I am okay to approximate the $g_{I^{(m)}}(\theta)$ via samples (but other methods are good too).

How could I proceed to obtain an (unbiased) estimate of the $g_I$ and without fully re-computing each $g_{I^{(m)}}$ separately?


Thoughts

  • If the $f_k$ were well-approximated by a parametric form (e.g., multivariate normal), I could estimate the parameters for each $f_k$ and then I would have a parametric form for any $g_I(\theta)$ too (as a product of "known" parametric densities).
  • Alternatively, I could approximate $f_k$ in a non-parametric form (e.g., KDE).
  • Note that even if such approximations to $f_k$ were okay, it seems that this procedure could be strongly biased due to uncertainty in the estimated parameters/samples.
  • Is there any way to compute/sample $g_I$ directly from the MCMC samples of $f_k$, $k \in I$? I am thinking of some sort of importance (re)sampling, although naïve methods might fail miserably if the $f_k$ overlap in regions with low probability density.
  • On the other hand, perhaps any method would fail miserably if the $f_k$ only overlap in regions with low probability density.
$\endgroup$

1 Answer 1

3
$\begingroup$

When it was posted, I missed this question, but it relates strongly to the current literature on scalable Monte Carlo algorithms. And in particular to the Bayesian processing of tall datasets, aiming at simulating posteriors that are the product of many functions that can each be turned into densities. Among many links of interest wrt this question, let me sample

While this is a link-only answer, it should give you a comfortable entry into the very active research in this area at the moment.

$\endgroup$
1
  • 1
    $\begingroup$ Thanks, excellent answer! (as usual) These links are very useful, as I wouldn't have known the correct names to search for. I just started browsing some of them, and I am glad to see that early papers (e.g., consensus MCMC) try out variants of what I had in mind. There's plenty to learn here. $\endgroup$
    – lacerbi
    Sep 30, 2016 at 2:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.