2
$\begingroup$

Suppose we have a sampler – eg a Monte Carlo sampler – for the posterior probability distribution $p(x \vert D)$ of a quantity $x=(y,z)$ consisting of a pair of continuous and multidimensional quantities. $D$ is a set of previously observed values of $x$.

Are there any general methods to obtain samples from the conditional distribution $p(y \vert z^*, D)$, for a given $z^*$, from the previous sampler or from samples of $x$ obtained therefrom?

(To be clearer, what I'm interested in is not just the conditional mean $\mathrm{E}(y \vert z^*, D)$ or similar statistics, but the full conditional distribution.)

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Related: Recent research shows that any estimate of the conditional mean of a response $Y$ given features $X$ will be rough and won't improve with more samples, unless $X$ can take on specific values with non-negligible probability or more assumptions hold (e.g., Lee and Barber 2021). $\endgroup$
    – Peter O.
    Jun 17 at 7:31
  • $\begingroup$ @PeterO. Very interesting work, thank you. I have the impression that the problem of "conditional sampling" (in the sense above) is somewhat neglected in the literature; but that can be lack of coverage on my part. Hence my question. $\endgroup$
    – pglpm
    Jun 17 at 7:34
  • $\begingroup$ @Xi'an Thank you, I'd be happy if you expanded your comment into an answer. My specific case is actually non-parametric inference (Dirichlet-process mixtures, and not just of normals), but I'd be interested in having a panoramic view of the general approaches. $\endgroup$
    – pglpm
    Jun 17 at 8:09
  • 1
    $\begingroup$ This paper on conditional Monte Carlo may prove related. $\endgroup$
    – Xi'an
    Jun 17 at 11:45
2
$\begingroup$

In the event the joint density $p(y,z|D)$ is available in closed form (possibly up to a multiplicative constant), a sample $x_1,\ldots,x_n$ can be exploited to produce a sample from $p(y|z^*,D)$ by

  1. acceptance-rejection, assuming $p(y,z^*|D)/p(y,z_i|D)$ is upper bounded [exact sampling, random sample size]
  2. sampling importance resampling, selecting with replacement $y_i$'s with probability proportional to $p(y,z^*|D)/p(y,z_i|D)$ [biased resampling, due to the renormalisation of the importance weights]
  3. Markov chain Monte Carlo, with, at iteration $t$, selecting one index i at random and replacing the current value $y^{(t)}$ by $y_i$ with Metropolis-Hastings probability$$1 \wedge \dfrac{p(y_i,z^*|D)}{p(y_i,z_i|D)} \dfrac{p(y^{(t)},z_i|D)}{p(y^{(t)},z^*|D)}$$[exact sampling assuming enough iterations for the Markov chain to reach stationarity]

Here is a basic R illustration

#generate conditional from joint
#x ~ Exp(1) & y|x ~ N(0,x)
#joint sample
N=1e5
smpl=cbind(z<-rexp(N),y<-z*rnorm(N))
#conditional on 
zo=2
we=dnorm(y,sd=zo)/dnorm(y,sd=z)
# 1. Accept-reject
yo=smpl[z>zo,2]
ya=yo*(runif(length(yo))<sqrt(zo/z[z>zo])*we[z>zo])
ya=ya[ya!=0]
# 2. sampling importance resampling
ess=round(sum(we)^2/sum(we^2)) #effective sample size
yi=sample(y,ess,prob=we,rep=TRUE)
# 3. MCMC
ym=rep(yi[1],N/10)
for (t in 2:length(ym)){
  i=sample(1:N,1)
  ym[t]=ifelse(runif(1)<we[i]*dnorm(ym[t-1],sd=z[i])/dnorm(ym[t-1],sd=zo),
               y[i],ym[t-1])}

Without such items of information, i.e. in a generative model when the sampler is a black box, there is no generic solution and approximate methods such as ABC or synthetic likelihood need be used.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for the informative summary! Could you add some references so I can dig further? Cheers! $\endgroup$
    – pglpm
    Jun 17 at 9:01
  • $\begingroup$ My references are regular Monte Carlo books like ours, where this specific issue is not mentioned. $\endgroup$
    – Xi'an
    Jun 17 at 9:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.