The problem
- Suppose I have $K$ independent probability densities $f_1(\theta), f_2(\theta), \ldots, f_K(\theta)$ defined over a variable $\theta$.
- I do not have a closed expression for the $f_k$, but they can be computed numerically up to a normalization constant (i.e., I can estimate them via MCMC sampling).
- My goal is to compute $g_I(\theta) = \frac{1}{Z_I} \prod_{k \in I} f_k(\theta)$, where $I \subseteq \{1,\ldots,K\}$ and $Z_I$ is a normalization constant. I am interested in computing $g_I(\theta)$ for several subsets $I^{(1)},\ldots,I^{(M)}$.
- I am okay to approximate the $g_{I^{(m)}}(\theta)$ via samples (but other methods are good too).
How could I proceed to obtain an (unbiased) estimate of the $g_I$ and without fully re-computing each $g_{I^{(m)}}$ separately?
Thoughts
- If the $f_k$ were well-approximated by a parametric form (e.g., multivariate normal), I could estimate the parameters for each $f_k$ and then I would have a parametric form for any $g_I(\theta)$ too (as a product of "known" parametric densities).
- Alternatively, I could approximate $f_k$ in a non-parametric form (e.g., KDE).
- Note that even if such approximations to $f_k$ were okay, it seems that this procedure could be strongly biased due to uncertainty in the estimated parameters/samples.
- Is there any way to compute/sample $g_I$ directly from the MCMC samples of $f_k$, $k \in I$? I am thinking of some sort of importance (re)sampling, although naïve methods might fail miserably if the $f_k$ overlap in regions with low probability density.
- On the other hand, perhaps any method would fail miserably if the $f_k$ only overlap in regions with low probability density.
