Sub-sampling a dataset to a different target distribution without replacement - bias correction? Suppose i have dataset $X$ and $Y$, and i want to sub-sample from $X$ so that the resulting (sample) distribution is as close to that of $Y$ as possible.
One thing i can do is subsample with probability (density) $\frac{d_y}{d_x}(X)$.
However, i notice that if i subsample without replacement some bias is introduced, whereas if i do so with replacement the resulting distribution is more similar to my target.
Question: Is there any way to sample from $X$ without replacement and still achieve a distribution that is similar to $Y$?
I have assumed the issue is sampling without replacement but if it is actually something else i am happy to be corrected.
Some code to illustrate the scenario:
set.seed(1235)

X=rexp(10000,rate=1)
Y=rexp(10000,rate=2)
mean(X);mean(Y)


## want to sub-sample from X to so that the resulting distribution is similar to that of Y
estimated_density=density(X,bw='SJ')
target_density=density(Y,bw='SJ')
d0=approx(estimated_density,xout=X)$y
d1=approx(target_density,xout=X)$y
d1=ifelse(is.na(d1),0,d1)


my_sub_sample=function(replace){
  summary(replicate(500,{
    subsample=sample(X,size = 5000,prob=d1/d0,replace = replace)
    mean(subsample)   
  }))

}

my_sub_sample(replace=T) ## mean close to 0.5
my_sub_sample(replace=F) ## mean not close to 0.5

 A: For arbitrary distributions, there isn't any way to ensure that a subsample of $X$ has a similar distribution to $Y$, whether you use replacement or not. Consider the distributions where $X \sim U(1,2)$ and $Y \sim U(3,4)$ - it's not even possible to draw a single value that would occur in both distributions, much less draw a sample from $X$ that is representative of $Y$.
The issue you have in your example is that even though the distributions are overlapping, your subsample is very large, at half the total population of $X$. Without replacement, a very large subsample of $X$ can only deviate so far from $X$, but how that deviates from $Y$ will require a direct comparison of the the $X$ and $Y$ distributions. But if you could do that, it's not clear why you'd bother subsampling $X$ instead of just drawing from $Y$ in the first place.
Suppose you want to draw from $X \sim N(0,1)$ to approximate $Y \sim N(5,1)$, with a population of $N=1000$. Only  a handful of samples from $X$ have values that are "useful" for representing $Y$, with values close to 5. If you sample without replacement, once those few samples from $X$ are "used up", you only have values that are highly inconsistent with $Y$ remaining. But there's no straightforward correction pattern, you'd have the same issue with opposite directionality if you were trying to approximate $Y \sim N(-5,1)$. Any fix will involve going back to the original $Y$ distribution to find what parts are being under-represented by the sample from $X$.
A: It looks like you are trying to use importance sampling, which is the technique that I would have suggested for this problem.
First, a note: your implementation is different to the usual way of implementing importance sampling.  Typically we would sample from a proposal distribution $x_i \sim q(\cdot)$ and weight the resulting samples by importance weights calculated by $p(x_i) / q(x_i)$, where $p(\cdot)$ is the target distribution.  Instead of sampling directly from the proposal, you are sampling in proportion to the importance weights (similar to this post).  I believe that is fine and shouldn't cause problems.
Unfortunately, there's a mismatch between the proposal distribution you used to do the sampling (resampling from $X$ without replacement) and what you used in the denominator of the importance weights (a density estimate of $X$ used in an iid fashion, i.e. assuming sampling with replacement).  I suspect it is the with/without replacement mismatch between these two parts that is the main cause of the bias.
If you could sample with replacement, then you shouldn't have any issues (as illustrated by your code).  I presume this is not an option for your particular application, otherwise you would have just done it that way.
If you want to sample without replacement, then one idea might be to adapt the denominator accordingly.  It needs to correctly specify the distribution of a set of values that are sampled without replacement from $X$.
I'm not sure if there's an easy way to do this.  You could try doing it iteratively as you sample each observation in turn (in your code above, you would need to re-run density() after drawing each element of X), but that would likely be computationally prohibitive.  There might be a better way, where you do a calculation for a whole sample from X all at once, but I can't immediately think of how to implement this.
By the way, if you only ever draw a small sample relative to the total population, the bias should be quite small too.
