# Monte Carlo or other general sampling approaches for conditional distributions?

Suppose we have a sampler – eg a Monte Carlo sampler – for the posterior probability distribution $$p(x \vert D)$$ of a quantity $$x=(y,z)$$ consisting of a pair of continuous and multidimensional quantities. $$D$$ is a set of previously observed values of $$x$$.

Are there any general methods to obtain samples from the conditional distribution $$p(y \vert z^*, D)$$, for a given $$z^*$$, from the previous sampler or from samples of $$x$$ obtained therefrom?

(To be clearer, what I'm interested in is not just the conditional mean $$\mathrm{E}(y \vert z^*, D)$$ or similar statistics, but the full conditional distribution.)

• Related: Recent research shows that any estimate of the conditional mean of a response $Y$ given features $X$ will be rough and won't improve with more samples, unless $X$ can take on specific values with non-negligible probability or more assumptions hold (e.g., Lee and Barber 2021). Jun 17, 2021 at 7:31
• @PeterO. Very interesting work, thank you. I have the impression that the problem of "conditional sampling" (in the sense above) is somewhat neglected in the literature; but that can be lack of coverage on my part. Hence my question. Jun 17, 2021 at 7:34
• @Xi'an Thank you, I'd be happy if you expanded your comment into an answer. My specific case is actually non-parametric inference (Dirichlet-process mixtures, and not just of normals), but I'd be interested in having a panoramic view of the general approaches. Jun 17, 2021 at 8:09
• This paper on conditional Monte Carlo may prove related. Jun 17, 2021 at 11:45

In the event the joint density $$p(y,z|D)$$ is available in closed form (possibly up to a multiplicative constant), a sample $$x_1,\ldots,x_n$$ can be exploited to produce a sample from $$p(y|z^*,D)$$ by

1. acceptance-rejection, assuming $$p(y,z^*|D)/p(y,z_i|D)$$ is upper bounded [exact sampling, random sample size]
2. sampling importance resampling, selecting with replacement $$y_i$$'s with probability proportional to $$p(y,z^*|D)/p(y,z_i|D)$$ [biased resampling, due to the renormalisation of the importance weights]
3. Markov chain Monte Carlo, with, at iteration $$t$$, selecting one index i at random and replacing the current value $$y^{(t)}$$ by $$y_i$$ with Metropolis-Hastings probability$$1 \wedge \dfrac{p(y_i,z^*|D)}{p(y_i,z_i|D)} \dfrac{p(y^{(t)},z_i|D)}{p(y^{(t)},z^*|D)}$$[exact sampling assuming enough iterations for the Markov chain to reach stationarity]

Here is a basic R illustration

#generate conditional from joint
#x ~ Exp(1) & y|x ~ N(0,x)
#joint sample
N=1e5
smpl=cbind(z<-rexp(N),y<-z*rnorm(N))
#conditional on
zo=2
we=dnorm(y,sd=zo)/dnorm(y,sd=z)
# 1. Accept-reject
yo=smpl[z>zo,2]
ya=yo*(runif(length(yo))<sqrt(zo/z[z>zo])*we[z>zo])
ya=ya[ya!=0]
# 2. sampling importance resampling
ess=round(sum(we)^2/sum(we^2)) #effective sample size
yi=sample(y,ess,prob=we,rep=TRUE)
# 3. MCMC
ym=rep(yi[1],N/10)
for (t in 2:length(ym)){
i=sample(1:N,1)
ym[t]=ifelse(runif(1)<we[i]*dnorm(ym[t-1],sd=z[i])/dnorm(ym[t-1],sd=zo),
y[i],ym[t-1])}


Without such items of information, i.e. in a generative model when the sampler is a black box, there is no generic solution and approximate methods such as ABC or synthetic likelihood need be used.

• Thank you for the informative summary! Could you add some references so I can dig further? Cheers! Jun 17, 2021 at 9:01
• My references are regular Monte Carlo books like ours, where this specific issue is not mentioned. Jun 17, 2021 at 9:09