# How to combine two sets of samples from an unnormalised distribution?

Imagine that I have an unnormalised distribution $P$ with density function $p(x)$ for $x \in \mathbb{R}^d$. $P$ has two well-separated modes and there are two sets of i.i.d. samples with the size $n$ that each of them has been taken from one of the modes. I would like to know how I can combine these samples in an unbiased way. In other words, let $\hat{\mu}_i$ be the empirical estimation of $\mu_i$, then I would like to find the $w_i$ that minimises the following error:

$\min \quad (\sum_{i}w_i(\hat{\mu}_i-\mu_i))^2$

Cheers

• Please explain more precisely how you can "sample ... from one of the modes." The language sounds like you might have a mixture. This is crucial because the relative amounts of the components in the mixture determine the solution, but you haven't provided any information about those amounts. – whuber Mar 24 '18 at 19:46
• Thanks, consider my problem as a stratified sampling that we do not know the weights of each strata. – KiaSh Mar 24 '18 at 19:52
• Then it would appear you have insufficient information to combine the results in any objective way. – whuber Mar 24 '18 at 22:26
• But I have $p(x)$; is not possible to use it to find the weights ? – KiaSh Mar 25 '18 at 13:03
• If you have all of $p$, then there's no question to be asked and your samples aren't needed: simply normalize $p$ to integrate to unity. – whuber Mar 25 '18 at 14:03

Here is a solution (see section 3.3) which uses the Renyi entropy:

https://arxiv.org/pdf/1806.03816.pdf

According to the paper, assuming that we have the Renyi entropy of the samples from each mode and the $p(x_i)$ for those samples, we can find the weight of each mode and the resulting estimator is asymptotically unbiased.

In practice, however, the problem/challenge is how to find the entropy: The authors suggested to use the Information Theoretical Estimators package [1] for estimating the entropy.

[1] Szabó, Zoltán. Information theoretical estimators toolbox. Journal of Machine Learning Research,15:283–287, 2014

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.