Can importance sampling be used as an actual sampling mechanism?

Question

This question is a duplicate of How can we use importance sampling for drawing posterior distribution samples? , but that question seems to lack additional detail and goes unanswered (for more than 2 years). So I am posting this question with more contexts.

Given a hard to sample $p(x)$, which can be e.g. a posterior, we can estimate its $E_{p}[x]$ by using $E_{q}\left[\frac{p(x)}{q(x)}x\right]$, where $q(x)$ is a easy to sample distribution, and has support that contains the support of $p$.

My question is, can we use a similar idea to sample from $p(x)$? Specifically, we can draw samples from $q(x)$, and let's call them $x_1, ..., x_n$. We can compute weight $w_1, ... w_n$ with $w_i=p(x_i)/q(x_i)$. Then we do a weighted sampling with replacement (bootstrapping?) from $x_1, ..., x_n$, with weight being $w_i$'s, i.e., we sample $y_i$ from $x_1, ..., x_n$ according the categorical distribution with weight $w_1, ..., w_n$. Then will those $y_i$ be distributed according to $p(x)$?

My intuitive concern is that $y_i$ will contain lots of duplicates, especially if the weights are not close to uniform. However, we are pretty much guaranteed that we will not encounter duplicate samples from any continuous distribution. So $y_i$ feels weird to me if they are distributed as $p(x)$.

Answer 1

The issue is less of having duplicates (MCMC runs also produce duplicates) than not being marginally distributed from $p$. While $$\mathbb E_q[h(Y) p(Y)/q(Y)]=\mathbb E_p[h(Y)]$$for any integrable function $h(\cdot)$, weighting and resampling an iid sample $(Y_1,\ldots,Y_n)$ from $q$ does not produce a sample from $p$, even marginally. The reason for the discrepancy is that the weighting-resampling step implies dividing the $p(Y_i)/q(Y_i)$ by the random sum of the weights, i.e., the index $i$ is selected with probability $$p(Y_i)/q(Y_i)\Big/\sum_j p(Y_j)/q(Y_j)$$ which modifies the marginal distribution of the resampled rv's, especially when the sum has an infinite variance.

Here is an illustration when $p$ is the density of a Student's $t_5$ distribution with mean 3 and $q$ is the density of a standard Normal distribution:

y=sample(x<-rnorm(1e7),re=TRUE,pr=dt(x-3,df=5)/dnorm(x))

which shows that the original Normal sample fails to cover the rhs of the tail of the Student's $t$ and hence that the weighted-resampled sample cannot recover.

Obviously, when the target $q$ has fatter tails than $p$, the method converges, as shown in this example:

y=sample(x<-rnorm(1e7),re=TRUE,pr=dnorm(x,2,.5)/dnorm(x))