2
$\begingroup$

This question is a duplicate of How can we use importance sampling for drawing posterior distribution samples? , but that question seems to lack additional detail and goes unanswered (for more than 2 years). So I am posting this question with more contexts.

Given a hard to sample $p(x)$, which can be e.g. a posterior, we can estimate its $E_{p}[x]$ by using $E_{q}\left[\frac{p(x)}{q(x)}x\right]$, where $q(x)$ is a easy to sample distribution, and has support that contains the support of $p$.

My question is, can we use a similar idea to sample from $p(x)$? Specifically, we can draw samples from $q(x)$, and let's call them $x_1, ..., x_n$. We can compute weight $w_1, ... w_n$ with $w_i=p(x_i)/q(x_i)$. Then we do a weighted sampling with replacement (bootstrapping?) from $x_1, ..., x_n$, with weight being $w_i$'s, i.e., we sample $y_i$ from $x_1, ..., x_n$ according the categorical distribution with weight $w_1, ..., w_n$. Then will those $y_i$ be distributed according to $p(x)$?

My intuitive concern is that $y_i$ will contain lots of duplicates, especially if the weights are not close to uniform. However, we are pretty much guaranteed that we will not encounter duplicate samples from any continuous distribution. So $y_i$ feels weird to me if they are distributed as $p(x)$.

$\endgroup$
  • 2
    $\begingroup$ yes, you're describing SIR (sampling importance resampling) aka factored sampling. It was invented in 1987! $\endgroup$ – Taylor Nov 22 '19 at 17:38
  • $\begingroup$ Reweighting simulations from $q(\cdot)$ by $p(x)/q(x)$ does not turn them into simulations from $p$, even marginally. $\endgroup$ – Xi'an Nov 23 '19 at 8:04
  • $\begingroup$ They are distributed according to p only in the limit as the number of particles (from which you are resampling one) goes to infinity. $\endgroup$ – Alex Lew Nov 26 '19 at 19:30
1
$\begingroup$

The issue is less of having duplicates (MCMC runs also produce duplicates) than not being marginally distributed from $p$. While $$\mathbb E_q[h(Y) p(Y)/q(Y)]=\mathbb E_p[h(Y)]$$for any integrable function $h(\cdot)$, weighting and resampling an iid sample $(Y_1,\ldots,Y_n)$ from $q$ does not produce a sample from $p$, even marginally. The reason for the discrepancy is that the weighting-resampling step implies dividing the $p(Y_i)/q(Y_i)$ by the random sum of the weights, i.e., the index $i$ is selected with probability $$p(Y_i)/q(Y_i)\Big/\sum_j p(Y_j)/q(Y_j)$$ which modifies the marginal distribution of the resampled rv's, especially when the sum has an infinite variance.

Here is an illustration when $p$ is the density of a Student's $t_5$ distribution with mean 3 and $q$ is the density of a standard Normal distribution:

y=sample(x<-rnorm(1e7),re=TRUE,pr=dt(x-3,df=5)/dnorm(x))

enter image description here

which shows that the original Normal sample fails to cover the rhs of the tail of the Student's $t$ and hence that the weighted-resampled sample cannot recover.

Obviously, when the target $q$ has fatter tails than $p$, the method converges, as shown in this example:

y=sample(x<-rnorm(1e7),re=TRUE,pr=dnorm(x,2,.5)/dnorm(x))

enter image description here

$\endgroup$
  • 1
    $\begingroup$ I am not quite sure if I follow this part: "dividing the $p(Y_i)/q(Y_i)$ by the random sum of the weights, which modifies the marginal distribution of the resampled rv's, especially when the sum has no variance". Could you explain in more detail? Specifically, why the marginal distribution is modified, and why the sum has no variance (isn't it a random variable, albeit with quite complicated pdf)? $\endgroup$ – Y.Z. Nov 26 '19 at 0:47
  • $\begingroup$ No variance means infinite variance. $\endgroup$ – Xi'an Nov 26 '19 at 7:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.